{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "967acdf8-b5f2-4746-a56f-28fdfc27595d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============GPU================\n",
      "Sat Dec 23 09:35:52 2023       \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
      "|-----------------------------------------+----------------------+----------------------+\n",
      "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                                         |                      |               MIG M. |\n",
      "|=========================================+======================+======================|\n",
      "|   0  NVIDIA GeForce RTX 3090        On  | 00000000:04:00.0 Off |                  N/A |\n",
      "| 30%   40C    P8              27W / 350W |      3MiB / 24576MiB |      0%      Default |\n",
      "|                                         |                      |                  N/A |\n",
      "+-----------------------------------------+----------------------+----------------------+\n",
      "|   1  NVIDIA GeForce RTX 3090        On  | 00000000:07:00.0 Off |                  N/A |\n",
      "|  0%   39C    P8              23W / 350W |      3MiB / 24576MiB |      0%      Default |\n",
      "|                                         |                      |                  N/A |\n",
      "+-----------------------------------------+----------------------+----------------------+\n",
      "|   2  NVIDIA GeForce RTX 3090        On  | 00000000:0A:00.0 Off |                  N/A |\n",
      "| 31%   41C    P8              21W / 350W |      3MiB / 24576MiB |      0%      Default |\n",
      "|                                         |                      |                  N/A |\n",
      "+-----------------------------------------+----------------------+----------------------+\n",
      "                                                                                         \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| Processes:                                                                            |\n",
      "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
      "|        ID   ID                                                             Usage      |\n",
      "|=======================================================================================|\n",
      "|  No running processes found                                                           |\n",
      "+---------------------------------------------------------------------------------------+\n",
      "============CUDA version================\n",
      "nvcc: NVIDIA (R) Cuda compiler driver\n",
      "Copyright (c) 2005-2022 NVIDIA Corporation\n",
      "Built on Wed_Sep_21_10:33:58_PDT_2022\n",
      "Cuda compilation tools, release 11.8, V11.8.89\n",
      "Build cuda_11.8.r11.8/compiler.31833905_0\n",
      "============CPU================\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "model name\t: Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz\n",
      "============Memory================\n",
      "MemTotal:       131851148 kB\n"
     ]
    }
   ],
   "source": [
    "# GPU\n",
    "print(\"============GPU================\")\n",
    "!nvidia-smi\n",
    "\n",
    "# CUDA version\n",
    "print(\"============CUDA version================\")\n",
    "!nvcc --version\n",
    "\n",
    "# CPU\n",
    "print(\"============CPU================\")\n",
    "!cat /proc/cpuinfo | grep model\\ name\n",
    "\n",
    "# Memory\n",
    "print(\"============Memory================\")\n",
    "!cat /proc/meminfo | grep MemTotal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bb0c71b2-b7d2-47b2-82ab-24619929a13d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'llama'...\n",
      "remote: Enumerating objects: 417, done.\u001b[K\n",
      "remote: Counting objects: 100% (71/71), done.\u001b[K\n",
      "remote: Compressing objects: 100% (49/49), done.\u001b[K\n",
      "remote: Total 417 (delta 29), reused 48 (delta 14), pack-reused 346\u001b[K\n",
      "Receiving objects: 100% (417/417), 1.10 MiB | 10.41 MiB/s, done.\n",
      "Resolving deltas: 100% (214/214), done.\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/facebookresearch/llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "080f46e7-5783-4fc1-9552-59f9021bdfc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/IPython/core/magics/osm.py:417: UserWarning: using dhist requires you to install the `pickleshare` library.\n",
      "  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]\n"
     ]
    }
   ],
   "source": [
    "%cd llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e764773d-63c6-4076-bc0b-79583e7927b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading LICENSE and Acceptable Usage Policy\n",
      "--2023-12-21 18:46:25--  https://download.llamameta.net/LICENSE?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.52, 108.138.106.87, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.52|:443... connected.\n",
      "HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable\n",
      "\n",
      "    The file is already fully retrieved; nothing to do.\n",
      "\n",
      "--2023-12-21 18:46:25--  https://download.llamameta.net/USE_POLICY.md?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.50, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable\n",
      "\n",
      "    The file is already fully retrieved; nothing to do.\n",
      "\n",
      "Downloading tokenizer\n",
      "--2023-12-21 18:46:25--  https://download.llamameta.net/tokenizer.model?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.52, 108.138.106.50, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 499723 (488K) [binary/octet-stream]\n",
      "Saving to: ‘./tokenizer.model’\n",
      "\n",
      "./tokenizer.model   100%[===================>] 488.01K  --.-KB/s    in 0.07s   \n",
      "\n",
      "2023-12-21 18:46:25 (7.11 MB/s) - ‘./tokenizer.model’ saved [499723/499723]\n",
      "\n",
      "--2023-12-21 18:46:25--  https://download.llamameta.net/tokenizer_checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.52, 108.138.106.50, 108.138.106.87, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.52|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 50 [binary/octet-stream]\n",
      "Saving to: ‘./tokenizer_checklist.chk’\n",
      "\n",
      "./tokenizer_checkli 100%[===================>]      50  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 18:46:26 (69.1 MB/s) - ‘./tokenizer_checklist.chk’ saved [50/50]\n",
      "\n",
      "tokenizer.model: OK\n",
      "Downloading llama-2-7b\n",
      "--2023-12-21 18:46:26--  https://download.llamameta.net/llama-2-7b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.52, 108.138.106.50, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.52|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13476925163 (13G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-7b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-7b/consol 100%[===================>]  12.55G  35.9MB/s    in 5m 13s  \n",
      "\n",
      "2023-12-21 18:51:39 (41.1 MB/s) - ‘./llama-2-7b/consolidated.00.pth’ saved [13476925163/13476925163]\n",
      "\n",
      "--2023-12-21 18:51:39--  https://download.llamameta.net/llama-2-7b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.87, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 102 [application/json]\n",
      "Saving to: ‘./llama-2-7b/params.json’\n",
      "\n",
      "./llama-2-7b/params 100%[===================>]     102  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 18:51:39 (10.1 MB/s) - ‘./llama-2-7b/params.json’ saved [102/102]\n",
      "\n",
      "--2023-12-21 18:51:39--  https://download.llamameta.net/llama-2-7b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.52, 108.138.106.50, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.52|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 100 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-7b/checklist.chk’\n",
      "\n",
      "./llama-2-7b/checkl 100%[===================>]     100  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 18:51:39 (119 MB/s) - ‘./llama-2-7b/checklist.chk’ saved [100/100]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "params.json: OK\n",
      "Downloading llama-2-13b\n",
      "--2023-12-21 18:52:04--  https://download.llamameta.net/llama-2-13b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.52, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13016329643 (12G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-13b/conso 100%[===================>]  12.12G  60.8MB/s    in 3m 59s  \n",
      "\n",
      "2023-12-21 18:56:03 (52.0 MB/s) - ‘./llama-2-13b/consolidated.00.pth’ saved [13016329643/13016329643]\n",
      "\n",
      "--2023-12-21 18:56:03--  https://download.llamameta.net/llama-2-13b/consolidated.01.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.87, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13016329643 (12G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/consolidated.01.pth’\n",
      "\n",
      "./llama-2-13b/conso 100%[===================>]  12.12G  71.7MB/s    in 4m 1s   \n",
      "\n",
      "2023-12-21 19:00:05 (51.6 MB/s) - ‘./llama-2-13b/consolidated.01.pth’ saved [13016329643/13016329643]\n",
      "\n",
      "--2023-12-21 19:00:05--  https://download.llamameta.net/llama-2-13b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.87, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 102 [application/json]\n",
      "Saving to: ‘./llama-2-13b/params.json’\n",
      "\n",
      "./llama-2-13b/param 100%[===================>]     102  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 19:00:05 (12.3 MB/s) - ‘./llama-2-13b/params.json’ saved [102/102]\n",
      "\n",
      "--2023-12-21 19:00:05--  https://download.llamameta.net/llama-2-13b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.23, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 154 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/checklist.chk’\n",
      "\n",
      "./llama-2-13b/check 100%[===================>]     154  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 19:00:05 (229 MB/s) - ‘./llama-2-13b/checklist.chk’ saved [154/154]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "consolidated.01.pth: OK\n",
      "params.json: OK\n",
      "Downloading llama-2-70b\n",
      "--2023-12-21 19:00:53--  https://download.llamameta.net/llama-2-70b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.23, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  99.3MB/s    in 3m 12s  \n",
      "\n",
      "2023-12-21 19:04:05 (85.8 MB/s) - ‘./llama-2-70b/consolidated.00.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:04:05--  https://download.llamameta.net/llama-2-70b/consolidated.01.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.23, 108.138.106.50, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.01.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  73.7MB/s    in 3m 28s  \n",
      "\n",
      "2023-12-21 19:07:33 (79.1 MB/s) - ‘./llama-2-70b/consolidated.01.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:07:33--  https://download.llamameta.net/llama-2-70b/consolidated.02.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.50, 108.138.106.23, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.50|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.02.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  97.8MB/s    in 3m 21s  \n",
      "\n",
      "2023-12-21 19:10:55 (81.7 MB/s) - ‘./llama-2-70b/consolidated.02.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:10:55--  https://download.llamameta.net/llama-2-70b/consolidated.03.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.23, 108.138.106.50, 108.138.106.87, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.03.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  88.7MB/s    in 3m 39s  \n",
      "\n",
      "2023-12-21 19:14:34 (75.0 MB/s) - ‘./llama-2-70b/consolidated.03.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:14:34--  https://download.llamameta.net/llama-2-70b/consolidated.04.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.23, 108.138.106.50, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.04.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  9.11MB/s    in 3m 59s  \n",
      "\n",
      "2023-12-21 19:18:34 (68.8 MB/s) - ‘./llama-2-70b/consolidated.04.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:18:34--  https://download.llamameta.net/llama-2-70b/consolidated.05.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.23, 108.138.106.50, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.05.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  91.5MB/s    in 3m 9s   \n",
      "\n",
      "2023-12-21 19:21:42 (87.2 MB/s) - ‘./llama-2-70b/consolidated.05.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:21:42--  https://download.llamameta.net/llama-2-70b/consolidated.06.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.52, 108.138.106.23, 108.138.106.50, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.52|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.06.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  88.5MB/s    in 3m 0s   \n",
      "\n",
      "2023-12-21 19:24:43 (91.3 MB/s) - ‘./llama-2-70b/consolidated.06.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:24:43--  https://download.llamameta.net/llama-2-70b/consolidated.07.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.23, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.07.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  46.3MB/s    in 4m 20s  \n",
      "\n",
      "2023-12-21 19:29:03 (63.2 MB/s) - ‘./llama-2-70b/consolidated.07.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-21 19:29:03--  https://download.llamameta.net/llama-2-70b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.23, 108.138.106.50, 108.138.106.52, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 147 [application/json]\n",
      "Saving to: ‘./llama-2-70b/params.json’\n",
      "\n",
      "./llama-2-70b/param 100%[===================>]     147  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 19:29:04 (1.68 MB/s) - ‘./llama-2-70b/params.json’ saved [147/147]\n",
      "\n",
      "--2023-12-21 19:29:04--  https://download.llamameta.net/llama-2-70b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoibW1ubGM4OTh2aHhtYjNwbnQzMWdvdmpzIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzI3MDU4M319fV19&Signature=aQwZ2GgPZgHodvWADnPipLid%7EgO-8Sp6j58tAz8XL6iEUuaIKCfSrSdQXImxh%7EiM3Jjj3K8THvK%7Et-V8Nzu3MOAzwc9-FGdQJpsjku8JsRQXGFLR3HdeeXcghSP1LP1nkB59XN4gcTxBeLcxcTsX%7Eo9qeeG5Nxe2oheb7HeRDCHr90Ur7HKxKWLBbczl%7Er6RjXMhipS05rE3pgsQaiur99Zm8dlaMbON2CfSG6OhhBHNy1BTG%7EgNIzExXCREHAOethodX8gm9uc8CzCr95k3%7EZ0wNHRiGcSxz77UhkJLGYF2euZilsR-wsCsv9wRgVyUNtfA4h5Z%7ESq59Vtozuq9uA__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=370498102131315\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 108.138.106.87, 108.138.106.52, 108.138.106.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|108.138.106.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 478 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/checklist.chk’\n",
      "\n",
      "./llama-2-70b/check 100%[===================>]     478  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-21 19:29:04 (38.1 MB/s) - ‘./llama-2-70b/checklist.chk’ saved [478/478]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "consolidated.01.pth: OK\n",
      "consolidated.02.pth: OK\n",
      "consolidated.03.pth: OK\n",
      "consolidated.04.pth: OK\n",
      "consolidated.05.pth: OK\n",
      "consolidated.06.pth: OK\n",
      "consolidated.07.pth: OK\n",
      "params.json: OK\n"
     ]
    }
   ],
   "source": [
    "# Define your PRESIGNED_URL and MODEL_SIZE in the script to prevent asking in the notebook\n",
    "!bash download.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1541dcc9-b822-4783-b6eb-020fc4a0316d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace\n"
     ]
    }
   ],
   "source": [
    "%cd /workspace\n",
    "!mkdir -p llama.cpp/models/7B-v2/\n",
    "!mv llama/llama-2-7b/* llama.cpp/models/7B-v2/\n",
    "!mkdir -p llama.cpp/models/13B-v2/\n",
    "!mv llama/llama-2-13b/* llama.cpp/models/13B-v2/\n",
    "!mkdir -p llama.cpp/models/70B-v2/\n",
    "!mv llama/llama-2-70b/* llama.cpp/models/70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cea2bab8-7c0d-42cc-8e32-064e71a58a74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp\n"
     ]
    }
   ],
   "source": [
    "%cd llama.cpp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9594719a-ea2a-4b8d-bd26-2c19f0a6a2de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 13283  100 13283    0     0  73817      0 --:--:-- --:--:-- --:--:-- 73794\n"
     ]
    }
   ],
   "source": [
    "# If you encounter the error \"does not appear to have a file named config.json\" when converting the models to ggml FP16 format, try to convert the model to huggingface format to get the config.json file.\n",
    "!curl -o convert_llama_weights_to_hf.py https://raw.githubusercontent.com/huggingface/transformers/main/src/transformers/models/llama/convert_llama_weights_to_hf.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "06b44ccf-4bf6-47a4-8f05-87a662822110",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp/models\n"
     ]
    }
   ],
   "source": [
    "%cd models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f541028-baaa-40f8-8e1c-c359b5ead34c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp tokenizer.model 7B-v2/\n",
    "!cp tokenizer.model 13B-v2/\n",
    "!cp tokenizer.model 70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fcc944e7-32a8-4918-8001-6b51fb835377",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp\n"
     ]
    }
   ],
   "source": [
    "%cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ae14bc36-f1ba-4069-82bc-63242471a393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting numpy==1.24.4 (from -r requirements.txt (line 1))\n",
      "  Using cached numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)\n",
      "Collecting sentencepiece==0.1.98 (from -r requirements.txt (line 2))\n",
      "  Using cached sentencepiece-0.1.98-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "Collecting transformers>=4.34.0 (from -r requirements.txt (line 3))\n",
      "  Using cached transformers-4.36.2-py3-none-any.whl.metadata (126 kB)\n",
      "Collecting gguf>=0.1.0 (from -r requirements.txt (line 4))\n",
      "  Using cached gguf-0.6.0-py3-none-any.whl.metadata (3.2 kB)\n",
      "Collecting protobuf>=4.21.0 (from -r requirements.txt (line 5))\n",
      "  Using cached protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (3.13.1)\n",
      "Collecting huggingface-hub<1.0,>=0.19.3 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Using cached huggingface_hub-0.20.1-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (6.0.1)\n",
      "Collecting regex!=2019.12.17 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Using cached regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (2.31.0)\n",
      "Collecting tokenizers<0.19,>=0.14 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Using cached tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
      "Collecting safetensors>=0.3.1 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Using cached safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
      "Collecting tqdm>=4.27 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Using cached tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers>=4.34.0->-r requirements.txt (line 3)) (2023.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers>=4.34.0->-r requirements.txt (line 3)) (4.8.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (2023.11.17)\n",
      "Using cached numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
      "Using cached transformers-4.36.2-py3-none-any.whl (8.2 MB)\n",
      "Using cached gguf-0.6.0-py3-none-any.whl (23 kB)\n",
      "Using cached protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl (294 kB)\n",
      "Using cached huggingface_hub-0.20.1-py3-none-any.whl (330 kB)\n",
      "Using cached regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (773 kB)\n",
      "Using cached safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "Using cached tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)\n",
      "Using cached tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
      "Installing collected packages: sentencepiece, tqdm, safetensors, regex, protobuf, numpy, huggingface-hub, gguf, tokenizers, transformers\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.26.2\n",
      "    Uninstalling numpy-1.26.2:\n",
      "      Successfully uninstalled numpy-1.26.2\n",
      "Successfully installed gguf-0.6.0 huggingface-hub-0.20.1 numpy-1.24.4 protobuf-4.25.1 regex-2023.10.3 safetensors-0.4.1 sentencepiece-0.1.98 tokenizers-0.15.0 tqdm-4.66.1 transformers-4.36.2\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3 -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# !pip uninstall accelerate # If you have this package, uninstall it first, then use `convert to hf model` to get the config.json.\n",
    "# install Python dependencies\n",
    "!python3 -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a95b3a38-17fb-4792-b1cd-4255e6ed2a7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/7B-v2/.\n",
      "Loading the checkpoint in a Llama model.\n",
      "Traceback (most recent call last):\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 319, in <module>\n",
      "    main()\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 307, in main\n",
      "    write_model(\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 271, in write_model\n",
      "    model = LlamaForCausalLM.from_pretrained(tmp_model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True)\n",
      "  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2863, in from_pretrained\n",
      "    raise ImportError(\n",
      "ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/13B-v2/.\n",
      "Loading the checkpoint in a Llama model.\n",
      "Traceback (most recent call last):\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 319, in <module>\n",
      "    main()\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 307, in main\n",
      "    write_model(\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 271, in write_model\n",
      "    model = LlamaForCausalLM.from_pretrained(tmp_model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True)\n",
      "  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2863, in from_pretrained\n",
      "    raise ImportError(\n",
      "ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/70B-v2/.\n"
     ]
    }
   ],
   "source": [
    "# We don't need these models actually. We only need this to figure out the config.json error.\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/7B-v2/ --model_size 7B --output_dir models/7B-v2/\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/13B-v2/ --model_size 13B --output_dir models/13B-v2/\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/70B-v2/ --model_size 70B --output_dir models/70B-v2/ # Surprisingly, it still solves the problem although you can't find the config.json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "839375fa-44f5-498c-8f1e-0e22ad8311ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit your params.json file if the \"vocab_size\" mismatch\n",
    "import json\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/7B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/7B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/13B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/13B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/70B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/70B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c5ce9c63-6f03-4736-a1df-56b9605f698b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model file models/7B-v2/consolidated.00.pth\n",
      "params = Params(n_vocab=32000, n_embd=4096, n_layer=32, n_ctx=4096, n_ff=11008, n_head=32, n_head_kv=32, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/7B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 4096]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [4096]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 4096]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [4096]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/7B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/291] Writing tensor token_embd.weight                      | size  32000 x   4096  | type F16  | T+   1\n",
      "[  2/291] Writing tensor output_norm.weight                     | size   4096           | type F32  | T+   1\n",
      "[  3/291] Writing tensor output.weight                          | size  32000 x   4096  | type F16  | T+   1\n",
      "[  4/291] Writing tensor blk.0.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  5/291] Writing tensor blk.0.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  6/291] Writing tensor blk.0.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  7/291] Writing tensor blk.0.attn_output.weight               | size   4096 x   4096  | type F16  | T+   1\n",
      "[  8/291] Writing tensor blk.0.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   1\n",
      "[  9/291] Writing tensor blk.0.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   1\n",
      "[ 10/291] Writing tensor blk.0.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   1\n",
      "[ 11/291] Writing tensor blk.0.attn_norm.weight                 | size   4096           | type F32  | T+   1\n",
      "[ 12/291] Writing tensor blk.0.ffn_norm.weight                  | size   4096           | type F32  | T+   1\n",
      "[ 13/291] Writing tensor blk.1.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 14/291] Writing tensor blk.1.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 15/291] Writing tensor blk.1.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 16/291] Writing tensor blk.1.attn_output.weight               | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 17/291] Writing tensor blk.1.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   1\n",
      "[ 18/291] Writing tensor blk.1.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   1\n",
      "[ 19/291] Writing tensor blk.1.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   1\n",
      "[ 20/291] Writing tensor blk.1.attn_norm.weight                 | size   4096           | type F32  | T+   2\n",
      "[ 21/291] Writing tensor blk.1.ffn_norm.weight                  | size   4096           | type F32  | T+   2\n",
      "[ 22/291] Writing tensor blk.2.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 23/291] Writing tensor blk.2.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 24/291] Writing tensor blk.2.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 25/291] Writing tensor blk.2.attn_output.weight               | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 26/291] Writing tensor blk.2.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 27/291] Writing tensor blk.2.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   2\n",
      "[ 28/291] Writing tensor blk.2.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 29/291] Writing tensor blk.2.attn_norm.weight                 | size   4096           | type F32  | T+   2\n",
      "[ 30/291] Writing tensor blk.2.ffn_norm.weight                  | size   4096           | type F32  | T+   2\n",
      "[ 31/291] Writing tensor blk.3.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 32/291] Writing tensor blk.3.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 33/291] Writing tensor blk.3.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 34/291] Writing tensor blk.3.attn_output.weight               | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 35/291] Writing tensor blk.3.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 36/291] Writing tensor blk.3.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   2\n",
      "[ 37/291] Writing tensor blk.3.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 38/291] Writing tensor blk.3.attn_norm.weight                 | size   4096           | type F32  | T+   2\n",
      "[ 39/291] Writing tensor blk.3.ffn_norm.weight                  | size   4096           | type F32  | T+   2\n",
      "[ 40/291] Writing tensor blk.4.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 41/291] Writing tensor blk.4.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 42/291] Writing tensor blk.4.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 43/291] Writing tensor blk.4.attn_output.weight               | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 44/291] Writing tensor blk.4.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 45/291] Writing tensor blk.4.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   3\n",
      "[ 46/291] Writing tensor blk.4.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 47/291] Writing tensor blk.4.attn_norm.weight                 | size   4096           | type F32  | T+   3\n",
      "[ 48/291] Writing tensor blk.4.ffn_norm.weight                  | size   4096           | type F32  | T+   3\n",
      "[ 49/291] Writing tensor blk.5.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 50/291] Writing tensor blk.5.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 51/291] Writing tensor blk.5.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 52/291] Writing tensor blk.5.attn_output.weight               | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 53/291] Writing tensor blk.5.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 54/291] Writing tensor blk.5.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   3\n",
      "[ 55/291] Writing tensor blk.5.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 56/291] Writing tensor blk.5.attn_norm.weight                 | size   4096           | type F32  | T+   3\n",
      "[ 57/291] Writing tensor blk.5.ffn_norm.weight                  | size   4096           | type F32  | T+   3\n",
      "[ 58/291] Writing tensor blk.6.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 59/291] Writing tensor blk.6.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 60/291] Writing tensor blk.6.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 61/291] Writing tensor blk.6.attn_output.weight               | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 62/291] Writing tensor blk.6.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 63/291] Writing tensor blk.6.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   3\n",
      "[ 64/291] Writing tensor blk.6.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 65/291] Writing tensor blk.6.attn_norm.weight                 | size   4096           | type F32  | T+   4\n",
      "[ 66/291] Writing tensor blk.6.ffn_norm.weight                  | size   4096           | type F32  | T+   4\n",
      "[ 67/291] Writing tensor blk.7.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 68/291] Writing tensor blk.7.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 69/291] Writing tensor blk.7.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 70/291] Writing tensor blk.7.attn_output.weight               | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 71/291] Writing tensor blk.7.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 72/291] Writing tensor blk.7.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   4\n",
      "[ 73/291] Writing tensor blk.7.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 74/291] Writing tensor blk.7.attn_norm.weight                 | size   4096           | type F32  | T+   4\n",
      "[ 75/291] Writing tensor blk.7.ffn_norm.weight                  | size   4096           | type F32  | T+   4\n",
      "[ 76/291] Writing tensor blk.8.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 77/291] Writing tensor blk.8.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 78/291] Writing tensor blk.8.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 79/291] Writing tensor blk.8.attn_output.weight               | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 80/291] Writing tensor blk.8.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 81/291] Writing tensor blk.8.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   4\n",
      "[ 82/291] Writing tensor blk.8.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 83/291] Writing tensor blk.8.attn_norm.weight                 | size   4096           | type F32  | T+   4\n",
      "[ 84/291] Writing tensor blk.8.ffn_norm.weight                  | size   4096           | type F32  | T+   4\n",
      "[ 85/291] Writing tensor blk.9.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 86/291] Writing tensor blk.9.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 87/291] Writing tensor blk.9.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 88/291] Writing tensor blk.9.attn_output.weight               | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 89/291] Writing tensor blk.9.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 90/291] Writing tensor blk.9.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   5\n",
      "[ 91/291] Writing tensor blk.9.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 92/291] Writing tensor blk.9.attn_norm.weight                 | size   4096           | type F32  | T+   5\n",
      "[ 93/291] Writing tensor blk.9.ffn_norm.weight                  | size   4096           | type F32  | T+   5\n",
      "[ 94/291] Writing tensor blk.10.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 95/291] Writing tensor blk.10.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 96/291] Writing tensor blk.10.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 97/291] Writing tensor blk.10.attn_output.weight              | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 98/291] Writing tensor blk.10.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 99/291] Writing tensor blk.10.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   5\n",
      "[100/291] Writing tensor blk.10.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   5\n",
      "[101/291] Writing tensor blk.10.attn_norm.weight                | size   4096           | type F32  | T+   5\n",
      "[102/291] Writing tensor blk.10.ffn_norm.weight                 | size   4096           | type F32  | T+   5\n",
      "[103/291] Writing tensor blk.11.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[104/291] Writing tensor blk.11.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[105/291] Writing tensor blk.11.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   5\n",
      "[106/291] Writing tensor blk.11.attn_output.weight              | size   4096 x   4096  | type F16  | T+   5\n",
      "[107/291] Writing tensor blk.11.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   5\n",
      "[108/291] Writing tensor blk.11.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   6\n",
      "[109/291] Writing tensor blk.11.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   6\n",
      "[110/291] Writing tensor blk.11.attn_norm.weight                | size   4096           | type F32  | T+   6\n",
      "[111/291] Writing tensor blk.11.ffn_norm.weight                 | size   4096           | type F32  | T+   6\n",
      "[112/291] Writing tensor blk.12.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[113/291] Writing tensor blk.12.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[114/291] Writing tensor blk.12.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[115/291] Writing tensor blk.12.attn_output.weight              | size   4096 x   4096  | type F16  | T+   6\n",
      "[116/291] Writing tensor blk.12.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   6\n",
      "[117/291] Writing tensor blk.12.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   6\n",
      "[118/291] Writing tensor blk.12.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   6\n",
      "[119/291] Writing tensor blk.12.attn_norm.weight                | size   4096           | type F32  | T+   6\n",
      "[120/291] Writing tensor blk.12.ffn_norm.weight                 | size   4096           | type F32  | T+   6\n",
      "[121/291] Writing tensor blk.13.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[122/291] Writing tensor blk.13.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[123/291] Writing tensor blk.13.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[124/291] Writing tensor blk.13.attn_output.weight              | size   4096 x   4096  | type F16  | T+   6\n",
      "[125/291] Writing tensor blk.13.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   6\n",
      "[126/291] Writing tensor blk.13.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   6\n",
      "[127/291] Writing tensor blk.13.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   6\n",
      "[128/291] Writing tensor blk.13.attn_norm.weight                | size   4096           | type F32  | T+   6\n",
      "[129/291] Writing tensor blk.13.ffn_norm.weight                 | size   4096           | type F32  | T+   6\n",
      "[130/291] Writing tensor blk.14.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[131/291] Writing tensor blk.14.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[132/291] Writing tensor blk.14.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[133/291] Writing tensor blk.14.attn_output.weight              | size   4096 x   4096  | type F16  | T+   6\n",
      "[134/291] Writing tensor blk.14.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   7\n",
      "[135/291] Writing tensor blk.14.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   7\n",
      "[136/291] Writing tensor blk.14.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   7\n",
      "[137/291] Writing tensor blk.14.attn_norm.weight                | size   4096           | type F32  | T+   7\n",
      "[138/291] Writing tensor blk.14.ffn_norm.weight                 | size   4096           | type F32  | T+   7\n",
      "[139/291] Writing tensor blk.15.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[140/291] Writing tensor blk.15.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[141/291] Writing tensor blk.15.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[142/291] Writing tensor blk.15.attn_output.weight              | size   4096 x   4096  | type F16  | T+   7\n",
      "[143/291] Writing tensor blk.15.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   7\n",
      "[144/291] Writing tensor blk.15.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   7\n",
      "[145/291] Writing tensor blk.15.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   7\n",
      "[146/291] Writing tensor blk.15.attn_norm.weight                | size   4096           | type F32  | T+   7\n",
      "[147/291] Writing tensor blk.15.ffn_norm.weight                 | size   4096           | type F32  | T+   7\n",
      "[148/291] Writing tensor blk.16.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[149/291] Writing tensor blk.16.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[150/291] Writing tensor blk.16.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[151/291] Writing tensor blk.16.attn_output.weight              | size   4096 x   4096  | type F16  | T+   7\n",
      "[152/291] Writing tensor blk.16.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   7\n",
      "[153/291] Writing tensor blk.16.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   8\n",
      "[154/291] Writing tensor blk.16.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   8\n",
      "[155/291] Writing tensor blk.16.attn_norm.weight                | size   4096           | type F32  | T+   8\n",
      "[156/291] Writing tensor blk.16.ffn_norm.weight                 | size   4096           | type F32  | T+   8\n",
      "[157/291] Writing tensor blk.17.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[158/291] Writing tensor blk.17.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[159/291] Writing tensor blk.17.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[160/291] Writing tensor blk.17.attn_output.weight              | size   4096 x   4096  | type F16  | T+   8\n",
      "[161/291] Writing tensor blk.17.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   8\n",
      "[162/291] Writing tensor blk.17.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   8\n",
      "[163/291] Writing tensor blk.17.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   8\n",
      "[164/291] Writing tensor blk.17.attn_norm.weight                | size   4096           | type F32  | T+   8\n",
      "[165/291] Writing tensor blk.17.ffn_norm.weight                 | size   4096           | type F32  | T+   8\n",
      "[166/291] Writing tensor blk.18.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[167/291] Writing tensor blk.18.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[168/291] Writing tensor blk.18.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[169/291] Writing tensor blk.18.attn_output.weight              | size   4096 x   4096  | type F16  | T+   8\n",
      "[170/291] Writing tensor blk.18.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   8\n",
      "[171/291] Writing tensor blk.18.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   8\n",
      "[172/291] Writing tensor blk.18.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   8\n",
      "[173/291] Writing tensor blk.18.attn_norm.weight                | size   4096           | type F32  | T+   8\n",
      "[174/291] Writing tensor blk.18.ffn_norm.weight                 | size   4096           | type F32  | T+   8\n",
      "[175/291] Writing tensor blk.19.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[176/291] Writing tensor blk.19.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[177/291] Writing tensor blk.19.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[178/291] Writing tensor blk.19.attn_output.weight              | size   4096 x   4096  | type F16  | T+   8\n",
      "[179/291] Writing tensor blk.19.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   9\n",
      "[180/291] Writing tensor blk.19.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   9\n",
      "[181/291] Writing tensor blk.19.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   9\n",
      "[182/291] Writing tensor blk.19.attn_norm.weight                | size   4096           | type F32  | T+   9\n",
      "[183/291] Writing tensor blk.19.ffn_norm.weight                 | size   4096           | type F32  | T+   9\n",
      "[184/291] Writing tensor blk.20.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[185/291] Writing tensor blk.20.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[186/291] Writing tensor blk.20.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[187/291] Writing tensor blk.20.attn_output.weight              | size   4096 x   4096  | type F16  | T+   9\n",
      "[188/291] Writing tensor blk.20.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   9\n",
      "[189/291] Writing tensor blk.20.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   9\n",
      "[190/291] Writing tensor blk.20.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   9\n",
      "[191/291] Writing tensor blk.20.attn_norm.weight                | size   4096           | type F32  | T+   9\n",
      "[192/291] Writing tensor blk.20.ffn_norm.weight                 | size   4096           | type F32  | T+   9\n",
      "[193/291] Writing tensor blk.21.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[194/291] Writing tensor blk.21.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[195/291] Writing tensor blk.21.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[196/291] Writing tensor blk.21.attn_output.weight              | size   4096 x   4096  | type F16  | T+   9\n",
      "[197/291] Writing tensor blk.21.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   9\n",
      "[198/291] Writing tensor blk.21.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  10\n",
      "[199/291] Writing tensor blk.21.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  10\n",
      "[200/291] Writing tensor blk.21.attn_norm.weight                | size   4096           | type F32  | T+  10\n",
      "[201/291] Writing tensor blk.21.ffn_norm.weight                 | size   4096           | type F32  | T+  10\n",
      "[202/291] Writing tensor blk.22.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[203/291] Writing tensor blk.22.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[204/291] Writing tensor blk.22.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[205/291] Writing tensor blk.22.attn_output.weight              | size   4096 x   4096  | type F16  | T+  10\n",
      "[206/291] Writing tensor blk.22.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  10\n",
      "[207/291] Writing tensor blk.22.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  10\n",
      "[208/291] Writing tensor blk.22.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  10\n",
      "[209/291] Writing tensor blk.22.attn_norm.weight                | size   4096           | type F32  | T+  10\n",
      "[210/291] Writing tensor blk.22.ffn_norm.weight                 | size   4096           | type F32  | T+  10\n",
      "[211/291] Writing tensor blk.23.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[212/291] Writing tensor blk.23.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[213/291] Writing tensor blk.23.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[214/291] Writing tensor blk.23.attn_output.weight              | size   4096 x   4096  | type F16  | T+  10\n",
      "[215/291] Writing tensor blk.23.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  10\n",
      "[216/291] Writing tensor blk.23.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  10\n",
      "[217/291] Writing tensor blk.23.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  10\n",
      "[218/291] Writing tensor blk.23.attn_norm.weight                | size   4096           | type F32  | T+  10\n",
      "[219/291] Writing tensor blk.23.ffn_norm.weight                 | size   4096           | type F32  | T+  10\n",
      "[220/291] Writing tensor blk.24.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[221/291] Writing tensor blk.24.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[222/291] Writing tensor blk.24.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[223/291] Writing tensor blk.24.attn_output.weight              | size   4096 x   4096  | type F16  | T+  11\n",
      "[224/291] Writing tensor blk.24.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  11\n",
      "[225/291] Writing tensor blk.24.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  11\n",
      "[226/291] Writing tensor blk.24.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  11\n",
      "[227/291] Writing tensor blk.24.attn_norm.weight                | size   4096           | type F32  | T+  11\n",
      "[228/291] Writing tensor blk.24.ffn_norm.weight                 | size   4096           | type F32  | T+  11\n",
      "[229/291] Writing tensor blk.25.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[230/291] Writing tensor blk.25.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[231/291] Writing tensor blk.25.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[232/291] Writing tensor blk.25.attn_output.weight              | size   4096 x   4096  | type F16  | T+  11\n",
      "[233/291] Writing tensor blk.25.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  11\n",
      "[234/291] Writing tensor blk.25.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  11\n",
      "[235/291] Writing tensor blk.25.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  11\n",
      "[236/291] Writing tensor blk.25.attn_norm.weight                | size   4096           | type F32  | T+  11\n",
      "[237/291] Writing tensor blk.25.ffn_norm.weight                 | size   4096           | type F32  | T+  11\n",
      "[238/291] Writing tensor blk.26.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[239/291] Writing tensor blk.26.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[240/291] Writing tensor blk.26.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[241/291] Writing tensor blk.26.attn_output.weight              | size   4096 x   4096  | type F16  | T+  11\n",
      "[242/291] Writing tensor blk.26.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  12\n",
      "[243/291] Writing tensor blk.26.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  12\n",
      "[244/291] Writing tensor blk.26.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  12\n",
      "[245/291] Writing tensor blk.26.attn_norm.weight                | size   4096           | type F32  | T+  12\n",
      "[246/291] Writing tensor blk.26.ffn_norm.weight                 | size   4096           | type F32  | T+  12\n",
      "[247/291] Writing tensor blk.27.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[248/291] Writing tensor blk.27.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[249/291] Writing tensor blk.27.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[250/291] Writing tensor blk.27.attn_output.weight              | size   4096 x   4096  | type F16  | T+  12\n",
      "[251/291] Writing tensor blk.27.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  12\n",
      "[252/291] Writing tensor blk.27.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  12\n",
      "[253/291] Writing tensor blk.27.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  12\n",
      "[254/291] Writing tensor blk.27.attn_norm.weight                | size   4096           | type F32  | T+  12\n",
      "[255/291] Writing tensor blk.27.ffn_norm.weight                 | size   4096           | type F32  | T+  12\n",
      "[256/291] Writing tensor blk.28.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[257/291] Writing tensor blk.28.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[258/291] Writing tensor blk.28.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[259/291] Writing tensor blk.28.attn_output.weight              | size   4096 x   4096  | type F16  | T+  12\n",
      "[260/291] Writing tensor blk.28.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  12\n",
      "[261/291] Writing tensor blk.28.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  12\n",
      "[262/291] Writing tensor blk.28.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  12\n",
      "[263/291] Writing tensor blk.28.attn_norm.weight                | size   4096           | type F32  | T+  13\n",
      "[264/291] Writing tensor blk.28.ffn_norm.weight                 | size   4096           | type F32  | T+  13\n",
      "[265/291] Writing tensor blk.29.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[266/291] Writing tensor blk.29.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[267/291] Writing tensor blk.29.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[268/291] Writing tensor blk.29.attn_output.weight              | size   4096 x   4096  | type F16  | T+  13\n",
      "[269/291] Writing tensor blk.29.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  13\n",
      "[270/291] Writing tensor blk.29.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  13\n",
      "[271/291] Writing tensor blk.29.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  13\n",
      "[272/291] Writing tensor blk.29.attn_norm.weight                | size   4096           | type F32  | T+  13\n",
      "[273/291] Writing tensor blk.29.ffn_norm.weight                 | size   4096           | type F32  | T+  13\n",
      "[274/291] Writing tensor blk.30.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[275/291] Writing tensor blk.30.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[276/291] Writing tensor blk.30.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[277/291] Writing tensor blk.30.attn_output.weight              | size   4096 x   4096  | type F16  | T+  13\n",
      "[278/291] Writing tensor blk.30.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  13\n",
      "[279/291] Writing tensor blk.30.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  13\n",
      "[280/291] Writing tensor blk.30.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  13\n",
      "[281/291] Writing tensor blk.30.attn_norm.weight                | size   4096           | type F32  | T+  13\n",
      "[282/291] Writing tensor blk.30.ffn_norm.weight                 | size   4096           | type F32  | T+  13\n",
      "[283/291] Writing tensor blk.31.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[284/291] Writing tensor blk.31.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[285/291] Writing tensor blk.31.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[286/291] Writing tensor blk.31.attn_output.weight              | size   4096 x   4096  | type F16  | T+  13\n",
      "[287/291] Writing tensor blk.31.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  14\n",
      "[288/291] Writing tensor blk.31.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  14\n",
      "[289/291] Writing tensor blk.31.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  14\n",
      "[290/291] Writing tensor blk.31.attn_norm.weight                | size   4096           | type F32  | T+  14\n",
      "[291/291] Writing tensor blk.31.ffn_norm.weight                 | size   4096           | type F32  | T+  14\n",
      "Wrote models/7B-v2/ggml-model-f16.gguf\n",
      "Loading model file models/13B-v2/consolidated.00.pth\n",
      "Loading model file models/13B-v2/consolidated.01.pth\n",
      "params = Params(n_vocab=32000, n_embd=5120, n_layer=40, n_ctx=4096, n_ff=13824, n_head=40, n_head_kv=40, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/13B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 5120]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [5120]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 5120]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.32.attention.wq.weight                    -> blk.32.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wk.weight                    -> blk.32.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wv.weight                    -> blk.32.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wo.weight                    -> blk.32.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.32.feed_forward.w1.weight                 -> blk.32.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.32.feed_forward.w2.weight                 -> blk.32.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.32.feed_forward.w3.weight                 -> blk.32.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.32.attention_norm.weight                  -> blk.32.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.32.ffn_norm.weight                        -> blk.32.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.33.attention.wq.weight                    -> blk.33.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wk.weight                    -> blk.33.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wv.weight                    -> blk.33.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wo.weight                    -> blk.33.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.33.feed_forward.w1.weight                 -> blk.33.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.33.feed_forward.w2.weight                 -> blk.33.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.33.feed_forward.w3.weight                 -> blk.33.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.33.attention_norm.weight                  -> blk.33.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.33.ffn_norm.weight                        -> blk.33.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.34.attention.wq.weight                    -> blk.34.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wk.weight                    -> blk.34.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wv.weight                    -> blk.34.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wo.weight                    -> blk.34.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.34.feed_forward.w1.weight                 -> blk.34.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.34.feed_forward.w2.weight                 -> blk.34.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.34.feed_forward.w3.weight                 -> blk.34.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.34.attention_norm.weight                  -> blk.34.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.34.ffn_norm.weight                        -> blk.34.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.35.attention.wq.weight                    -> blk.35.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wk.weight                    -> blk.35.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wv.weight                    -> blk.35.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wo.weight                    -> blk.35.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.35.feed_forward.w1.weight                 -> blk.35.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.35.feed_forward.w2.weight                 -> blk.35.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.35.feed_forward.w3.weight                 -> blk.35.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.35.attention_norm.weight                  -> blk.35.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.35.ffn_norm.weight                        -> blk.35.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.36.attention.wq.weight                    -> blk.36.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wk.weight                    -> blk.36.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wv.weight                    -> blk.36.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wo.weight                    -> blk.36.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.36.feed_forward.w1.weight                 -> blk.36.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.36.feed_forward.w2.weight                 -> blk.36.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.36.feed_forward.w3.weight                 -> blk.36.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.36.attention_norm.weight                  -> blk.36.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.36.ffn_norm.weight                        -> blk.36.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.37.attention.wq.weight                    -> blk.37.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wk.weight                    -> blk.37.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wv.weight                    -> blk.37.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wo.weight                    -> blk.37.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.37.feed_forward.w1.weight                 -> blk.37.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.37.feed_forward.w2.weight                 -> blk.37.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.37.feed_forward.w3.weight                 -> blk.37.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.37.attention_norm.weight                  -> blk.37.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.37.ffn_norm.weight                        -> blk.37.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.38.attention.wq.weight                    -> blk.38.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wk.weight                    -> blk.38.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wv.weight                    -> blk.38.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wo.weight                    -> blk.38.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.38.feed_forward.w1.weight                 -> blk.38.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.38.feed_forward.w2.weight                 -> blk.38.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.38.feed_forward.w3.weight                 -> blk.38.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.38.attention_norm.weight                  -> blk.38.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.38.ffn_norm.weight                        -> blk.38.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.39.attention.wq.weight                    -> blk.39.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wk.weight                    -> blk.39.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wv.weight                    -> blk.39.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wo.weight                    -> blk.39.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.39.feed_forward.w1.weight                 -> blk.39.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.39.feed_forward.w2.weight                 -> blk.39.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.39.feed_forward.w3.weight                 -> blk.39.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.39.attention_norm.weight                  -> blk.39.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.39.ffn_norm.weight                        -> blk.39.ffn_norm.weight                   | BF16   | [5120]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/13B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/363] Writing tensor token_embd.weight                      | size  32000 x   5120  | type F16  | T+   1\n",
      "[  2/363] Writing tensor output_norm.weight                     | size   5120           | type F32  | T+   1\n",
      "[  3/363] Writing tensor output.weight                          | size  32000 x   5120  | type F16  | T+   1\n",
      "[  4/363] Writing tensor blk.0.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   1\n",
      "[  5/363] Writing tensor blk.0.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   1\n",
      "[  6/363] Writing tensor blk.0.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   1\n",
      "[  7/363] Writing tensor blk.0.attn_output.weight               | size   5120 x   5120  | type F16  | T+   1\n",
      "[  8/363] Writing tensor blk.0.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   1\n",
      "[  9/363] Writing tensor blk.0.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   2\n",
      "[ 10/363] Writing tensor blk.0.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   2\n",
      "[ 11/363] Writing tensor blk.0.attn_norm.weight                 | size   5120           | type F32  | T+   2\n",
      "[ 12/363] Writing tensor blk.0.ffn_norm.weight                  | size   5120           | type F32  | T+   2\n",
      "[ 13/363] Writing tensor blk.1.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 14/363] Writing tensor blk.1.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 15/363] Writing tensor blk.1.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 16/363] Writing tensor blk.1.attn_output.weight               | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 17/363] Writing tensor blk.1.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   2\n",
      "[ 18/363] Writing tensor blk.1.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   3\n",
      "[ 19/363] Writing tensor blk.1.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   3\n",
      "[ 20/363] Writing tensor blk.1.attn_norm.weight                 | size   5120           | type F32  | T+   3\n",
      "[ 21/363] Writing tensor blk.1.ffn_norm.weight                  | size   5120           | type F32  | T+   3\n",
      "[ 22/363] Writing tensor blk.2.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   3\n",
      "[ 23/363] Writing tensor blk.2.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   3\n",
      "[ 24/363] Writing tensor blk.2.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   3\n",
      "[ 25/363] Writing tensor blk.2.attn_output.weight               | size   5120 x   5120  | type F16  | T+   3\n",
      "[ 26/363] Writing tensor blk.2.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   4\n",
      "[ 27/363] Writing tensor blk.2.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   4\n",
      "[ 28/363] Writing tensor blk.2.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   5\n",
      "[ 29/363] Writing tensor blk.2.attn_norm.weight                 | size   5120           | type F32  | T+   5\n",
      "[ 30/363] Writing tensor blk.2.ffn_norm.weight                  | size   5120           | type F32  | T+   5\n",
      "[ 31/363] Writing tensor blk.3.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 32/363] Writing tensor blk.3.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 33/363] Writing tensor blk.3.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 34/363] Writing tensor blk.3.attn_output.weight               | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 35/363] Writing tensor blk.3.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   5\n",
      "[ 36/363] Writing tensor blk.3.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   6\n",
      "[ 37/363] Writing tensor blk.3.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   6\n",
      "[ 38/363] Writing tensor blk.3.attn_norm.weight                 | size   5120           | type F32  | T+   6\n",
      "[ 39/363] Writing tensor blk.3.ffn_norm.weight                  | size   5120           | type F32  | T+   6\n",
      "[ 40/363] Writing tensor blk.4.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   6\n",
      "[ 41/363] Writing tensor blk.4.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   6\n",
      "[ 42/363] Writing tensor blk.4.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   6\n",
      "[ 43/363] Writing tensor blk.4.attn_output.weight               | size   5120 x   5120  | type F16  | T+   6\n",
      "[ 44/363] Writing tensor blk.4.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   6\n",
      "[ 45/363] Writing tensor blk.4.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   7\n",
      "[ 46/363] Writing tensor blk.4.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   7\n",
      "[ 47/363] Writing tensor blk.4.attn_norm.weight                 | size   5120           | type F32  | T+   7\n",
      "[ 48/363] Writing tensor blk.4.ffn_norm.weight                  | size   5120           | type F32  | T+   7\n",
      "[ 49/363] Writing tensor blk.5.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 50/363] Writing tensor blk.5.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 51/363] Writing tensor blk.5.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 52/363] Writing tensor blk.5.attn_output.weight               | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 53/363] Writing tensor blk.5.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   8\n",
      "[ 54/363] Writing tensor blk.5.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   8\n",
      "[ 55/363] Writing tensor blk.5.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   8\n",
      "[ 56/363] Writing tensor blk.5.attn_norm.weight                 | size   5120           | type F32  | T+   8\n",
      "[ 57/363] Writing tensor blk.5.ffn_norm.weight                  | size   5120           | type F32  | T+   8\n",
      "[ 58/363] Writing tensor blk.6.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 59/363] Writing tensor blk.6.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 60/363] Writing tensor blk.6.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 61/363] Writing tensor blk.6.attn_output.weight               | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 62/363] Writing tensor blk.6.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   9\n",
      "[ 63/363] Writing tensor blk.6.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   9\n",
      "[ 64/363] Writing tensor blk.6.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   9\n",
      "[ 65/363] Writing tensor blk.6.attn_norm.weight                 | size   5120           | type F32  | T+   9\n",
      "[ 66/363] Writing tensor blk.6.ffn_norm.weight                  | size   5120           | type F32  | T+   9\n",
      "[ 67/363] Writing tensor blk.7.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   9\n",
      "[ 68/363] Writing tensor blk.7.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  10\n",
      "[ 69/363] Writing tensor blk.7.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  10\n",
      "[ 70/363] Writing tensor blk.7.attn_output.weight               | size   5120 x   5120  | type F16  | T+  10\n",
      "[ 71/363] Writing tensor blk.7.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  10\n",
      "[ 72/363] Writing tensor blk.7.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  10\n",
      "[ 73/363] Writing tensor blk.7.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  11\n",
      "[ 74/363] Writing tensor blk.7.attn_norm.weight                 | size   5120           | type F32  | T+  11\n",
      "[ 75/363] Writing tensor blk.7.ffn_norm.weight                  | size   5120           | type F32  | T+  11\n",
      "[ 76/363] Writing tensor blk.8.attn_q.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 77/363] Writing tensor blk.8.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 78/363] Writing tensor blk.8.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 79/363] Writing tensor blk.8.attn_output.weight               | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 80/363] Writing tensor blk.8.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  11\n",
      "[ 81/363] Writing tensor blk.8.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  12\n",
      "[ 82/363] Writing tensor blk.8.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  12\n",
      "[ 83/363] Writing tensor blk.8.attn_norm.weight                 | size   5120           | type F32  | T+  12\n",
      "[ 84/363] Writing tensor blk.8.ffn_norm.weight                  | size   5120           | type F32  | T+  12\n",
      "[ 85/363] Writing tensor blk.9.attn_q.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 86/363] Writing tensor blk.9.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 87/363] Writing tensor blk.9.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 88/363] Writing tensor blk.9.attn_output.weight               | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 89/363] Writing tensor blk.9.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  12\n",
      "[ 90/363] Writing tensor blk.9.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  13\n",
      "[ 91/363] Writing tensor blk.9.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  13\n",
      "[ 92/363] Writing tensor blk.9.attn_norm.weight                 | size   5120           | type F32  | T+  13\n",
      "[ 93/363] Writing tensor blk.9.ffn_norm.weight                  | size   5120           | type F32  | T+  13\n",
      "[ 94/363] Writing tensor blk.10.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 95/363] Writing tensor blk.10.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 96/363] Writing tensor blk.10.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 97/363] Writing tensor blk.10.attn_output.weight              | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 98/363] Writing tensor blk.10.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  14\n",
      "[ 99/363] Writing tensor blk.10.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  14\n",
      "[100/363] Writing tensor blk.10.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  14\n",
      "[101/363] Writing tensor blk.10.attn_norm.weight                | size   5120           | type F32  | T+  14\n",
      "[102/363] Writing tensor blk.10.ffn_norm.weight                 | size   5120           | type F32  | T+  14\n",
      "[103/363] Writing tensor blk.11.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  14\n",
      "[104/363] Writing tensor blk.11.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  14\n",
      "[105/363] Writing tensor blk.11.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  14\n",
      "[106/363] Writing tensor blk.11.attn_output.weight              | size   5120 x   5120  | type F16  | T+  14\n",
      "[107/363] Writing tensor blk.11.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  15\n",
      "[108/363] Writing tensor blk.11.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  15\n",
      "[109/363] Writing tensor blk.11.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  16\n",
      "[110/363] Writing tensor blk.11.attn_norm.weight                | size   5120           | type F32  | T+  16\n",
      "[111/363] Writing tensor blk.11.ffn_norm.weight                 | size   5120           | type F32  | T+  16\n",
      "[112/363] Writing tensor blk.12.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[113/363] Writing tensor blk.12.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[114/363] Writing tensor blk.12.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[115/363] Writing tensor blk.12.attn_output.weight              | size   5120 x   5120  | type F16  | T+  16\n",
      "[116/363] Writing tensor blk.12.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  16\n",
      "[117/363] Writing tensor blk.12.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  17\n",
      "[118/363] Writing tensor blk.12.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  17\n",
      "[119/363] Writing tensor blk.12.attn_norm.weight                | size   5120           | type F32  | T+  17\n",
      "[120/363] Writing tensor blk.12.ffn_norm.weight                 | size   5120           | type F32  | T+  17\n",
      "[121/363] Writing tensor blk.13.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  17\n",
      "[122/363] Writing tensor blk.13.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  17\n",
      "[123/363] Writing tensor blk.13.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  17\n",
      "[124/363] Writing tensor blk.13.attn_output.weight              | size   5120 x   5120  | type F16  | T+  17\n",
      "[125/363] Writing tensor blk.13.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  17\n",
      "[126/363] Writing tensor blk.13.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  18\n",
      "[127/363] Writing tensor blk.13.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  18\n",
      "[128/363] Writing tensor blk.13.attn_norm.weight                | size   5120           | type F32  | T+  18\n",
      "[129/363] Writing tensor blk.13.ffn_norm.weight                 | size   5120           | type F32  | T+  18\n",
      "[130/363] Writing tensor blk.14.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  18\n",
      "[131/363] Writing tensor blk.14.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  18\n",
      "[132/363] Writing tensor blk.14.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  18\n",
      "[133/363] Writing tensor blk.14.attn_output.weight              | size   5120 x   5120  | type F16  | T+  18\n",
      "[134/363] Writing tensor blk.14.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  19\n",
      "[135/363] Writing tensor blk.14.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  19\n",
      "[136/363] Writing tensor blk.14.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  19\n",
      "[137/363] Writing tensor blk.14.attn_norm.weight                | size   5120           | type F32  | T+  19\n",
      "[138/363] Writing tensor blk.14.ffn_norm.weight                 | size   5120           | type F32  | T+  19\n",
      "[139/363] Writing tensor blk.15.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[140/363] Writing tensor blk.15.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[141/363] Writing tensor blk.15.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[142/363] Writing tensor blk.15.attn_output.weight              | size   5120 x   5120  | type F16  | T+  19\n",
      "[143/363] Writing tensor blk.15.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  20\n",
      "[144/363] Writing tensor blk.15.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  20\n",
      "[145/363] Writing tensor blk.15.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  21\n",
      "[146/363] Writing tensor blk.15.attn_norm.weight                | size   5120           | type F32  | T+  21\n",
      "[147/363] Writing tensor blk.15.ffn_norm.weight                 | size   5120           | type F32  | T+  21\n",
      "[148/363] Writing tensor blk.16.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  21\n",
      "[149/363] Writing tensor blk.16.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  21\n",
      "[150/363] Writing tensor blk.16.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  21\n",
      "[151/363] Writing tensor blk.16.attn_output.weight              | size   5120 x   5120  | type F16  | T+  21\n",
      "[152/363] Writing tensor blk.16.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  21\n",
      "[153/363] Writing tensor blk.16.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  22\n",
      "[154/363] Writing tensor blk.16.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  22\n",
      "[155/363] Writing tensor blk.16.attn_norm.weight                | size   5120           | type F32  | T+  22\n",
      "[156/363] Writing tensor blk.16.ffn_norm.weight                 | size   5120           | type F32  | T+  22\n",
      "[157/363] Writing tensor blk.17.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[158/363] Writing tensor blk.17.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[159/363] Writing tensor blk.17.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[160/363] Writing tensor blk.17.attn_output.weight              | size   5120 x   5120  | type F16  | T+  22\n",
      "[161/363] Writing tensor blk.17.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  22\n",
      "[162/363] Writing tensor blk.17.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  23\n",
      "[163/363] Writing tensor blk.17.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  23\n",
      "[164/363] Writing tensor blk.17.attn_norm.weight                | size   5120           | type F32  | T+  23\n",
      "[165/363] Writing tensor blk.17.ffn_norm.weight                 | size   5120           | type F32  | T+  23\n",
      "[166/363] Writing tensor blk.18.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[167/363] Writing tensor blk.18.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[168/363] Writing tensor blk.18.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[169/363] Writing tensor blk.18.attn_output.weight              | size   5120 x   5120  | type F16  | T+  23\n",
      "[170/363] Writing tensor blk.18.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  24\n",
      "[171/363] Writing tensor blk.18.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  24\n",
      "[172/363] Writing tensor blk.18.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  24\n",
      "[173/363] Writing tensor blk.18.attn_norm.weight                | size   5120           | type F32  | T+  24\n",
      "[174/363] Writing tensor blk.18.ffn_norm.weight                 | size   5120           | type F32  | T+  24\n",
      "[175/363] Writing tensor blk.19.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  24\n",
      "[176/363] Writing tensor blk.19.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  24\n",
      "[177/363] Writing tensor blk.19.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  24\n",
      "[178/363] Writing tensor blk.19.attn_output.weight              | size   5120 x   5120  | type F16  | T+  24\n",
      "[179/363] Writing tensor blk.19.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  25\n",
      "[180/363] Writing tensor blk.19.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  25\n",
      "[181/363] Writing tensor blk.19.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  26\n",
      "[182/363] Writing tensor blk.19.attn_norm.weight                | size   5120           | type F32  | T+  26\n",
      "[183/363] Writing tensor blk.19.ffn_norm.weight                 | size   5120           | type F32  | T+  26\n",
      "[184/363] Writing tensor blk.20.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[185/363] Writing tensor blk.20.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[186/363] Writing tensor blk.20.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[187/363] Writing tensor blk.20.attn_output.weight              | size   5120 x   5120  | type F16  | T+  26\n",
      "[188/363] Writing tensor blk.20.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  26\n",
      "[189/363] Writing tensor blk.20.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  27\n",
      "[190/363] Writing tensor blk.20.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  27\n",
      "[191/363] Writing tensor blk.20.attn_norm.weight                | size   5120           | type F32  | T+  27\n",
      "[192/363] Writing tensor blk.20.ffn_norm.weight                 | size   5120           | type F32  | T+  27\n",
      "[193/363] Writing tensor blk.21.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[194/363] Writing tensor blk.21.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[195/363] Writing tensor blk.21.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[196/363] Writing tensor blk.21.attn_output.weight              | size   5120 x   5120  | type F16  | T+  27\n",
      "[197/363] Writing tensor blk.21.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  27\n",
      "[198/363] Writing tensor blk.21.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  28\n",
      "[199/363] Writing tensor blk.21.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  28\n",
      "[200/363] Writing tensor blk.21.attn_norm.weight                | size   5120           | type F32  | T+  28\n",
      "[201/363] Writing tensor blk.21.ffn_norm.weight                 | size   5120           | type F32  | T+  28\n",
      "[202/363] Writing tensor blk.22.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  28\n",
      "[203/363] Writing tensor blk.22.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  28\n",
      "[204/363] Writing tensor blk.22.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  28\n",
      "[205/363] Writing tensor blk.22.attn_output.weight              | size   5120 x   5120  | type F16  | T+  28\n",
      "[206/363] Writing tensor blk.22.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  29\n",
      "[207/363] Writing tensor blk.22.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  29\n",
      "[208/363] Writing tensor blk.22.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  29\n",
      "[209/363] Writing tensor blk.22.attn_norm.weight                | size   5120           | type F32  | T+  29\n",
      "[210/363] Writing tensor blk.22.ffn_norm.weight                 | size   5120           | type F32  | T+  29\n",
      "[211/363] Writing tensor blk.23.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[212/363] Writing tensor blk.23.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[213/363] Writing tensor blk.23.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[214/363] Writing tensor blk.23.attn_output.weight              | size   5120 x   5120  | type F16  | T+  29\n",
      "[215/363] Writing tensor blk.23.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  30\n",
      "[216/363] Writing tensor blk.23.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  30\n",
      "[217/363] Writing tensor blk.23.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  30\n",
      "[218/363] Writing tensor blk.23.attn_norm.weight                | size   5120           | type F32  | T+  30\n",
      "[219/363] Writing tensor blk.23.ffn_norm.weight                 | size   5120           | type F32  | T+  30\n",
      "[220/363] Writing tensor blk.24.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  30\n",
      "[221/363] Writing tensor blk.24.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  31\n",
      "[222/363] Writing tensor blk.24.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  31\n",
      "[223/363] Writing tensor blk.24.attn_output.weight              | size   5120 x   5120  | type F16  | T+  31\n",
      "[224/363] Writing tensor blk.24.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  31\n",
      "[225/363] Writing tensor blk.24.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  31\n",
      "[226/363] Writing tensor blk.24.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  32\n",
      "[227/363] Writing tensor blk.24.attn_norm.weight                | size   5120           | type F32  | T+  32\n",
      "[228/363] Writing tensor blk.24.ffn_norm.weight                 | size   5120           | type F32  | T+  32\n",
      "[229/363] Writing tensor blk.25.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  32\n",
      "[230/363] Writing tensor blk.25.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  32\n",
      "[231/363] Writing tensor blk.25.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  32\n",
      "[232/363] Writing tensor blk.25.attn_output.weight              | size   5120 x   5120  | type F16  | T+  32\n",
      "[233/363] Writing tensor blk.25.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  32\n",
      "[234/363] Writing tensor blk.25.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  33\n",
      "[235/363] Writing tensor blk.25.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  33\n",
      "[236/363] Writing tensor blk.25.attn_norm.weight                | size   5120           | type F32  | T+  33\n",
      "[237/363] Writing tensor blk.25.ffn_norm.weight                 | size   5120           | type F32  | T+  33\n",
      "[238/363] Writing tensor blk.26.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  33\n",
      "[239/363] Writing tensor blk.26.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  33\n",
      "[240/363] Writing tensor blk.26.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  33\n",
      "[241/363] Writing tensor blk.26.attn_output.weight              | size   5120 x   5120  | type F16  | T+  33\n",
      "[242/363] Writing tensor blk.26.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  33\n",
      "[243/363] Writing tensor blk.26.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  34\n",
      "[244/363] Writing tensor blk.26.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  34\n",
      "[245/363] Writing tensor blk.26.attn_norm.weight                | size   5120           | type F32  | T+  34\n",
      "[246/363] Writing tensor blk.26.ffn_norm.weight                 | size   5120           | type F32  | T+  34\n",
      "[247/363] Writing tensor blk.27.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  34\n",
      "[248/363] Writing tensor blk.27.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  34\n",
      "[249/363] Writing tensor blk.27.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  34\n",
      "[250/363] Writing tensor blk.27.attn_output.weight              | size   5120 x   5120  | type F16  | T+  34\n",
      "[251/363] Writing tensor blk.27.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  35\n",
      "[252/363] Writing tensor blk.27.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  35\n",
      "[253/363] Writing tensor blk.27.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  35\n",
      "[254/363] Writing tensor blk.27.attn_norm.weight                | size   5120           | type F32  | T+  35\n",
      "[255/363] Writing tensor blk.27.ffn_norm.weight                 | size   5120           | type F32  | T+  35\n",
      "[256/363] Writing tensor blk.28.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[257/363] Writing tensor blk.28.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[258/363] Writing tensor blk.28.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[259/363] Writing tensor blk.28.attn_output.weight              | size   5120 x   5120  | type F16  | T+  35\n",
      "[260/363] Writing tensor blk.28.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  36\n",
      "[261/363] Writing tensor blk.28.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  36\n",
      "[262/363] Writing tensor blk.28.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  36\n",
      "[263/363] Writing tensor blk.28.attn_norm.weight                | size   5120           | type F32  | T+  37\n",
      "[264/363] Writing tensor blk.28.ffn_norm.weight                 | size   5120           | type F32  | T+  37\n",
      "[265/363] Writing tensor blk.29.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[266/363] Writing tensor blk.29.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[267/363] Writing tensor blk.29.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[268/363] Writing tensor blk.29.attn_output.weight              | size   5120 x   5120  | type F16  | T+  37\n",
      "[269/363] Writing tensor blk.29.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  37\n",
      "[270/363] Writing tensor blk.29.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  37\n",
      "[271/363] Writing tensor blk.29.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  38\n",
      "[272/363] Writing tensor blk.29.attn_norm.weight                | size   5120           | type F32  | T+  38\n",
      "[273/363] Writing tensor blk.29.ffn_norm.weight                 | size   5120           | type F32  | T+  38\n",
      "[274/363] Writing tensor blk.30.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[275/363] Writing tensor blk.30.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[276/363] Writing tensor blk.30.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[277/363] Writing tensor blk.30.attn_output.weight              | size   5120 x   5120  | type F16  | T+  38\n",
      "[278/363] Writing tensor blk.30.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  38\n",
      "[279/363] Writing tensor blk.30.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  39\n",
      "[280/363] Writing tensor blk.30.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  39\n",
      "[281/363] Writing tensor blk.30.attn_norm.weight                | size   5120           | type F32  | T+  39\n",
      "[282/363] Writing tensor blk.30.ffn_norm.weight                 | size   5120           | type F32  | T+  39\n",
      "[283/363] Writing tensor blk.31.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  39\n",
      "[284/363] Writing tensor blk.31.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  39\n",
      "[285/363] Writing tensor blk.31.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  39\n",
      "[286/363] Writing tensor blk.31.attn_output.weight              | size   5120 x   5120  | type F16  | T+  39\n",
      "[287/363] Writing tensor blk.31.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  39\n",
      "[288/363] Writing tensor blk.31.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  40\n",
      "[289/363] Writing tensor blk.31.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  40\n",
      "[290/363] Writing tensor blk.31.attn_norm.weight                | size   5120           | type F32  | T+  40\n",
      "[291/363] Writing tensor blk.31.ffn_norm.weight                 | size   5120           | type F32  | T+  40\n",
      "[292/363] Writing tensor blk.32.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[293/363] Writing tensor blk.32.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[294/363] Writing tensor blk.32.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[295/363] Writing tensor blk.32.attn_output.weight              | size   5120 x   5120  | type F16  | T+  40\n",
      "[296/363] Writing tensor blk.32.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  41\n",
      "[297/363] Writing tensor blk.32.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  41\n",
      "[298/363] Writing tensor blk.32.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  41\n",
      "[299/363] Writing tensor blk.32.attn_norm.weight                | size   5120           | type F32  | T+  41\n",
      "[300/363] Writing tensor blk.32.ffn_norm.weight                 | size   5120           | type F32  | T+  41\n",
      "[301/363] Writing tensor blk.33.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[302/363] Writing tensor blk.33.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[303/363] Writing tensor blk.33.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[304/363] Writing tensor blk.33.attn_output.weight              | size   5120 x   5120  | type F16  | T+  41\n",
      "[305/363] Writing tensor blk.33.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  42\n",
      "[306/363] Writing tensor blk.33.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  42\n",
      "[307/363] Writing tensor blk.33.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  43\n",
      "[308/363] Writing tensor blk.33.attn_norm.weight                | size   5120           | type F32  | T+  43\n",
      "[309/363] Writing tensor blk.33.ffn_norm.weight                 | size   5120           | type F32  | T+  43\n",
      "[310/363] Writing tensor blk.34.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[311/363] Writing tensor blk.34.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[312/363] Writing tensor blk.34.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[313/363] Writing tensor blk.34.attn_output.weight              | size   5120 x   5120  | type F16  | T+  43\n",
      "[314/363] Writing tensor blk.34.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  43\n",
      "[315/363] Writing tensor blk.34.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  43\n",
      "[316/363] Writing tensor blk.34.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  44\n",
      "[317/363] Writing tensor blk.34.attn_norm.weight                | size   5120           | type F32  | T+  44\n",
      "[318/363] Writing tensor blk.34.ffn_norm.weight                 | size   5120           | type F32  | T+  44\n",
      "[319/363] Writing tensor blk.35.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[320/363] Writing tensor blk.35.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[321/363] Writing tensor blk.35.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[322/363] Writing tensor blk.35.attn_output.weight              | size   5120 x   5120  | type F16  | T+  44\n",
      "[323/363] Writing tensor blk.35.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  44\n",
      "[324/363] Writing tensor blk.35.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  45\n",
      "[325/363] Writing tensor blk.35.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  45\n",
      "[326/363] Writing tensor blk.35.attn_norm.weight                | size   5120           | type F32  | T+  45\n",
      "[327/363] Writing tensor blk.35.ffn_norm.weight                 | size   5120           | type F32  | T+  45\n",
      "[328/363] Writing tensor blk.36.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[329/363] Writing tensor blk.36.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[330/363] Writing tensor blk.36.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[331/363] Writing tensor blk.36.attn_output.weight              | size   5120 x   5120  | type F16  | T+  45\n",
      "[332/363] Writing tensor blk.36.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  45\n",
      "[333/363] Writing tensor blk.36.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  46\n",
      "[334/363] Writing tensor blk.36.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  46\n",
      "[335/363] Writing tensor blk.36.attn_norm.weight                | size   5120           | type F32  | T+  46\n",
      "[336/363] Writing tensor blk.36.ffn_norm.weight                 | size   5120           | type F32  | T+  46\n",
      "[337/363] Writing tensor blk.37.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  46\n",
      "[338/363] Writing tensor blk.37.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  46\n",
      "[339/363] Writing tensor blk.37.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  46\n",
      "[340/363] Writing tensor blk.37.attn_output.weight              | size   5120 x   5120  | type F16  | T+  46\n",
      "[341/363] Writing tensor blk.37.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  47\n",
      "[342/363] Writing tensor blk.37.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  47\n",
      "[343/363] Writing tensor blk.37.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  47\n",
      "[344/363] Writing tensor blk.37.attn_norm.weight                | size   5120           | type F32  | T+  47\n",
      "[345/363] Writing tensor blk.37.ffn_norm.weight                 | size   5120           | type F32  | T+  47\n",
      "[346/363] Writing tensor blk.38.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  47\n",
      "[347/363] Writing tensor blk.38.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  47\n",
      "[348/363] Writing tensor blk.38.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  48\n",
      "[349/363] Writing tensor blk.38.attn_output.weight              | size   5120 x   5120  | type F16  | T+  48\n",
      "[350/363] Writing tensor blk.38.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  48\n",
      "[351/363] Writing tensor blk.38.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  48\n",
      "[352/363] Writing tensor blk.38.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  49\n",
      "[353/363] Writing tensor blk.38.attn_norm.weight                | size   5120           | type F32  | T+  49\n",
      "[354/363] Writing tensor blk.38.ffn_norm.weight                 | size   5120           | type F32  | T+  49\n",
      "[355/363] Writing tensor blk.39.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  49\n",
      "[356/363] Writing tensor blk.39.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  49\n",
      "[357/363] Writing tensor blk.39.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  49\n",
      "[358/363] Writing tensor blk.39.attn_output.weight              | size   5120 x   5120  | type F16  | T+  49\n",
      "[359/363] Writing tensor blk.39.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  49\n",
      "[360/363] Writing tensor blk.39.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  50\n",
      "[361/363] Writing tensor blk.39.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  50\n",
      "[362/363] Writing tensor blk.39.attn_norm.weight                | size   5120           | type F32  | T+  50\n",
      "[363/363] Writing tensor blk.39.ffn_norm.weight                 | size   5120           | type F32  | T+  50\n",
      "Wrote models/13B-v2/ggml-model-f16.gguf\n",
      "Loading model file models/70B-v2/consolidated.00.pth\n",
      "Loading model file models/70B-v2/consolidated.01.pth\n",
      "Loading model file models/70B-v2/consolidated.02.pth\n",
      "Loading model file models/70B-v2/consolidated.03.pth\n",
      "Loading model file models/70B-v2/consolidated.04.pth\n",
      "Loading model file models/70B-v2/consolidated.05.pth\n",
      "Loading model file models/70B-v2/consolidated.06.pth\n",
      "Loading model file models/70B-v2/consolidated.07.pth\n",
      "params = Params(n_vocab=32000, n_embd=8192, n_layer=80, n_ctx=4096, n_ff=28672, n_head=64, n_head_kv=8, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/70B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 8192]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [8192]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 8192]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.32.attention.wq.weight                    -> blk.32.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.32.attention.wk.weight                    -> blk.32.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.32.attention.wv.weight                    -> blk.32.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.32.attention.wo.weight                    -> blk.32.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.32.feed_forward.w1.weight                 -> blk.32.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.32.feed_forward.w2.weight                 -> blk.32.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.32.feed_forward.w3.weight                 -> blk.32.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.32.attention_norm.weight                  -> blk.32.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.32.ffn_norm.weight                        -> blk.32.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.33.attention.wq.weight                    -> blk.33.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.33.attention.wk.weight                    -> blk.33.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.33.attention.wv.weight                    -> blk.33.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.33.attention.wo.weight                    -> blk.33.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.33.feed_forward.w1.weight                 -> blk.33.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.33.feed_forward.w2.weight                 -> blk.33.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.33.feed_forward.w3.weight                 -> blk.33.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.33.attention_norm.weight                  -> blk.33.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.33.ffn_norm.weight                        -> blk.33.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.34.attention.wq.weight                    -> blk.34.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.34.attention.wk.weight                    -> blk.34.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.34.attention.wv.weight                    -> blk.34.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.34.attention.wo.weight                    -> blk.34.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.34.feed_forward.w1.weight                 -> blk.34.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.34.feed_forward.w2.weight                 -> blk.34.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.34.feed_forward.w3.weight                 -> blk.34.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.34.attention_norm.weight                  -> blk.34.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.34.ffn_norm.weight                        -> blk.34.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.35.attention.wq.weight                    -> blk.35.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.35.attention.wk.weight                    -> blk.35.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.35.attention.wv.weight                    -> blk.35.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.35.attention.wo.weight                    -> blk.35.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.35.feed_forward.w1.weight                 -> blk.35.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.35.feed_forward.w2.weight                 -> blk.35.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.35.feed_forward.w3.weight                 -> blk.35.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.35.attention_norm.weight                  -> blk.35.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.35.ffn_norm.weight                        -> blk.35.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.36.attention.wq.weight                    -> blk.36.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.36.attention.wk.weight                    -> blk.36.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.36.attention.wv.weight                    -> blk.36.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.36.attention.wo.weight                    -> blk.36.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.36.feed_forward.w1.weight                 -> blk.36.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.36.feed_forward.w2.weight                 -> blk.36.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.36.feed_forward.w3.weight                 -> blk.36.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.36.attention_norm.weight                  -> blk.36.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.36.ffn_norm.weight                        -> blk.36.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.37.attention.wq.weight                    -> blk.37.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.37.attention.wk.weight                    -> blk.37.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.37.attention.wv.weight                    -> blk.37.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.37.attention.wo.weight                    -> blk.37.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.37.feed_forward.w1.weight                 -> blk.37.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.37.feed_forward.w2.weight                 -> blk.37.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.37.feed_forward.w3.weight                 -> blk.37.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.37.attention_norm.weight                  -> blk.37.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.37.ffn_norm.weight                        -> blk.37.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.38.attention.wq.weight                    -> blk.38.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.38.attention.wk.weight                    -> blk.38.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.38.attention.wv.weight                    -> blk.38.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.38.attention.wo.weight                    -> blk.38.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.38.feed_forward.w1.weight                 -> blk.38.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.38.feed_forward.w2.weight                 -> blk.38.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.38.feed_forward.w3.weight                 -> blk.38.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.38.attention_norm.weight                  -> blk.38.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.38.ffn_norm.weight                        -> blk.38.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.39.attention.wq.weight                    -> blk.39.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.39.attention.wk.weight                    -> blk.39.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.39.attention.wv.weight                    -> blk.39.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.39.attention.wo.weight                    -> blk.39.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.39.feed_forward.w1.weight                 -> blk.39.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.39.feed_forward.w2.weight                 -> blk.39.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.39.feed_forward.w3.weight                 -> blk.39.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.39.attention_norm.weight                  -> blk.39.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.39.ffn_norm.weight                        -> blk.39.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.40.attention.wq.weight                    -> blk.40.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.40.attention.wk.weight                    -> blk.40.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.40.attention.wv.weight                    -> blk.40.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.40.attention.wo.weight                    -> blk.40.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.40.feed_forward.w1.weight                 -> blk.40.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.40.feed_forward.w2.weight                 -> blk.40.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.40.feed_forward.w3.weight                 -> blk.40.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.40.attention_norm.weight                  -> blk.40.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.40.ffn_norm.weight                        -> blk.40.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.41.attention.wq.weight                    -> blk.41.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.41.attention.wk.weight                    -> blk.41.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.41.attention.wv.weight                    -> blk.41.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.41.attention.wo.weight                    -> blk.41.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.41.feed_forward.w1.weight                 -> blk.41.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.41.feed_forward.w2.weight                 -> blk.41.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.41.feed_forward.w3.weight                 -> blk.41.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.41.attention_norm.weight                  -> blk.41.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.41.ffn_norm.weight                        -> blk.41.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.42.attention.wq.weight                    -> blk.42.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.42.attention.wk.weight                    -> blk.42.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.42.attention.wv.weight                    -> blk.42.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.42.attention.wo.weight                    -> blk.42.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.42.feed_forward.w1.weight                 -> blk.42.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.42.feed_forward.w2.weight                 -> blk.42.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.42.feed_forward.w3.weight                 -> blk.42.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.42.attention_norm.weight                  -> blk.42.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.42.ffn_norm.weight                        -> blk.42.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.43.attention.wq.weight                    -> blk.43.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.43.attention.wk.weight                    -> blk.43.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.43.attention.wv.weight                    -> blk.43.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.43.attention.wo.weight                    -> blk.43.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.43.feed_forward.w1.weight                 -> blk.43.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.43.feed_forward.w2.weight                 -> blk.43.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.43.feed_forward.w3.weight                 -> blk.43.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.43.attention_norm.weight                  -> blk.43.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.43.ffn_norm.weight                        -> blk.43.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.44.attention.wq.weight                    -> blk.44.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.44.attention.wk.weight                    -> blk.44.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.44.attention.wv.weight                    -> blk.44.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.44.attention.wo.weight                    -> blk.44.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.44.feed_forward.w1.weight                 -> blk.44.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.44.feed_forward.w2.weight                 -> blk.44.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.44.feed_forward.w3.weight                 -> blk.44.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.44.attention_norm.weight                  -> blk.44.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.44.ffn_norm.weight                        -> blk.44.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.45.attention.wq.weight                    -> blk.45.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.45.attention.wk.weight                    -> blk.45.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.45.attention.wv.weight                    -> blk.45.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.45.attention.wo.weight                    -> blk.45.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.45.feed_forward.w1.weight                 -> blk.45.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.45.feed_forward.w2.weight                 -> blk.45.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.45.feed_forward.w3.weight                 -> blk.45.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.45.attention_norm.weight                  -> blk.45.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.45.ffn_norm.weight                        -> blk.45.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.46.attention.wq.weight                    -> blk.46.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.46.attention.wk.weight                    -> blk.46.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.46.attention.wv.weight                    -> blk.46.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.46.attention.wo.weight                    -> blk.46.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.46.feed_forward.w1.weight                 -> blk.46.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.46.feed_forward.w2.weight                 -> blk.46.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.46.feed_forward.w3.weight                 -> blk.46.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.46.attention_norm.weight                  -> blk.46.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.46.ffn_norm.weight                        -> blk.46.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.47.attention.wq.weight                    -> blk.47.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.47.attention.wk.weight                    -> blk.47.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.47.attention.wv.weight                    -> blk.47.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.47.attention.wo.weight                    -> blk.47.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.47.feed_forward.w1.weight                 -> blk.47.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.47.feed_forward.w2.weight                 -> blk.47.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.47.feed_forward.w3.weight                 -> blk.47.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.47.attention_norm.weight                  -> blk.47.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.47.ffn_norm.weight                        -> blk.47.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.48.attention.wq.weight                    -> blk.48.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.48.attention.wk.weight                    -> blk.48.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.48.attention.wv.weight                    -> blk.48.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.48.attention.wo.weight                    -> blk.48.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.48.feed_forward.w1.weight                 -> blk.48.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.48.feed_forward.w2.weight                 -> blk.48.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.48.feed_forward.w3.weight                 -> blk.48.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.48.attention_norm.weight                  -> blk.48.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.48.ffn_norm.weight                        -> blk.48.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.49.attention.wq.weight                    -> blk.49.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.49.attention.wk.weight                    -> blk.49.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.49.attention.wv.weight                    -> blk.49.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.49.attention.wo.weight                    -> blk.49.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.49.feed_forward.w1.weight                 -> blk.49.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.49.feed_forward.w2.weight                 -> blk.49.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.49.feed_forward.w3.weight                 -> blk.49.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.49.attention_norm.weight                  -> blk.49.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.49.ffn_norm.weight                        -> blk.49.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.50.attention.wq.weight                    -> blk.50.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.50.attention.wk.weight                    -> blk.50.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.50.attention.wv.weight                    -> blk.50.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.50.attention.wo.weight                    -> blk.50.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.50.feed_forward.w1.weight                 -> blk.50.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.50.feed_forward.w2.weight                 -> blk.50.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.50.feed_forward.w3.weight                 -> blk.50.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.50.attention_norm.weight                  -> blk.50.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.50.ffn_norm.weight                        -> blk.50.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.51.attention.wq.weight                    -> blk.51.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.51.attention.wk.weight                    -> blk.51.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.51.attention.wv.weight                    -> blk.51.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.51.attention.wo.weight                    -> blk.51.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.51.feed_forward.w1.weight                 -> blk.51.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.51.feed_forward.w2.weight                 -> blk.51.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.51.feed_forward.w3.weight                 -> blk.51.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.51.attention_norm.weight                  -> blk.51.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.51.ffn_norm.weight                        -> blk.51.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.52.attention.wq.weight                    -> blk.52.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.52.attention.wk.weight                    -> blk.52.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.52.attention.wv.weight                    -> blk.52.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.52.attention.wo.weight                    -> blk.52.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.52.feed_forward.w1.weight                 -> blk.52.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.52.feed_forward.w2.weight                 -> blk.52.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.52.feed_forward.w3.weight                 -> blk.52.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.52.attention_norm.weight                  -> blk.52.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.52.ffn_norm.weight                        -> blk.52.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.53.attention.wq.weight                    -> blk.53.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.53.attention.wk.weight                    -> blk.53.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.53.attention.wv.weight                    -> blk.53.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.53.attention.wo.weight                    -> blk.53.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.53.feed_forward.w1.weight                 -> blk.53.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.53.feed_forward.w2.weight                 -> blk.53.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.53.feed_forward.w3.weight                 -> blk.53.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.53.attention_norm.weight                  -> blk.53.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.53.ffn_norm.weight                        -> blk.53.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.54.attention.wq.weight                    -> blk.54.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.54.attention.wk.weight                    -> blk.54.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.54.attention.wv.weight                    -> blk.54.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.54.attention.wo.weight                    -> blk.54.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.54.feed_forward.w1.weight                 -> blk.54.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.54.feed_forward.w2.weight                 -> blk.54.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.54.feed_forward.w3.weight                 -> blk.54.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.54.attention_norm.weight                  -> blk.54.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.54.ffn_norm.weight                        -> blk.54.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.55.attention.wq.weight                    -> blk.55.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.55.attention.wk.weight                    -> blk.55.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.55.attention.wv.weight                    -> blk.55.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.55.attention.wo.weight                    -> blk.55.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.55.feed_forward.w1.weight                 -> blk.55.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.55.feed_forward.w2.weight                 -> blk.55.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.55.feed_forward.w3.weight                 -> blk.55.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.55.attention_norm.weight                  -> blk.55.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.55.ffn_norm.weight                        -> blk.55.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.56.attention.wq.weight                    -> blk.56.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.56.attention.wk.weight                    -> blk.56.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.56.attention.wv.weight                    -> blk.56.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.56.attention.wo.weight                    -> blk.56.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.56.feed_forward.w1.weight                 -> blk.56.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.56.feed_forward.w2.weight                 -> blk.56.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.56.feed_forward.w3.weight                 -> blk.56.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.56.attention_norm.weight                  -> blk.56.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.56.ffn_norm.weight                        -> blk.56.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.57.attention.wq.weight                    -> blk.57.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.57.attention.wk.weight                    -> blk.57.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.57.attention.wv.weight                    -> blk.57.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.57.attention.wo.weight                    -> blk.57.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.57.feed_forward.w1.weight                 -> blk.57.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.57.feed_forward.w2.weight                 -> blk.57.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.57.feed_forward.w3.weight                 -> blk.57.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.57.attention_norm.weight                  -> blk.57.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.57.ffn_norm.weight                        -> blk.57.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.58.attention.wq.weight                    -> blk.58.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.58.attention.wk.weight                    -> blk.58.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.58.attention.wv.weight                    -> blk.58.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.58.attention.wo.weight                    -> blk.58.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.58.feed_forward.w1.weight                 -> blk.58.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.58.feed_forward.w2.weight                 -> blk.58.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.58.feed_forward.w3.weight                 -> blk.58.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.58.attention_norm.weight                  -> blk.58.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.58.ffn_norm.weight                        -> blk.58.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.59.attention.wq.weight                    -> blk.59.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.59.attention.wk.weight                    -> blk.59.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.59.attention.wv.weight                    -> blk.59.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.59.attention.wo.weight                    -> blk.59.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.59.feed_forward.w1.weight                 -> blk.59.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.59.feed_forward.w2.weight                 -> blk.59.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.59.feed_forward.w3.weight                 -> blk.59.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.59.attention_norm.weight                  -> blk.59.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.59.ffn_norm.weight                        -> blk.59.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.60.attention.wq.weight                    -> blk.60.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.60.attention.wk.weight                    -> blk.60.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.60.attention.wv.weight                    -> blk.60.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.60.attention.wo.weight                    -> blk.60.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.60.feed_forward.w1.weight                 -> blk.60.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.60.feed_forward.w2.weight                 -> blk.60.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.60.feed_forward.w3.weight                 -> blk.60.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.60.attention_norm.weight                  -> blk.60.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.60.ffn_norm.weight                        -> blk.60.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.61.attention.wq.weight                    -> blk.61.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.61.attention.wk.weight                    -> blk.61.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.61.attention.wv.weight                    -> blk.61.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.61.attention.wo.weight                    -> blk.61.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.61.feed_forward.w1.weight                 -> blk.61.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.61.feed_forward.w2.weight                 -> blk.61.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.61.feed_forward.w3.weight                 -> blk.61.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.61.attention_norm.weight                  -> blk.61.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.61.ffn_norm.weight                        -> blk.61.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.62.attention.wq.weight                    -> blk.62.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.62.attention.wk.weight                    -> blk.62.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.62.attention.wv.weight                    -> blk.62.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.62.attention.wo.weight                    -> blk.62.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.62.feed_forward.w1.weight                 -> blk.62.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.62.feed_forward.w2.weight                 -> blk.62.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.62.feed_forward.w3.weight                 -> blk.62.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.62.attention_norm.weight                  -> blk.62.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.62.ffn_norm.weight                        -> blk.62.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.63.attention.wq.weight                    -> blk.63.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.63.attention.wk.weight                    -> blk.63.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.63.attention.wv.weight                    -> blk.63.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.63.attention.wo.weight                    -> blk.63.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.63.feed_forward.w1.weight                 -> blk.63.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.63.feed_forward.w2.weight                 -> blk.63.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.63.feed_forward.w3.weight                 -> blk.63.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.63.attention_norm.weight                  -> blk.63.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.63.ffn_norm.weight                        -> blk.63.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.64.attention.wq.weight                    -> blk.64.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.64.attention.wk.weight                    -> blk.64.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.64.attention.wv.weight                    -> blk.64.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.64.attention.wo.weight                    -> blk.64.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.64.feed_forward.w1.weight                 -> blk.64.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.64.feed_forward.w2.weight                 -> blk.64.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.64.feed_forward.w3.weight                 -> blk.64.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.64.attention_norm.weight                  -> blk.64.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.64.ffn_norm.weight                        -> blk.64.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.65.attention.wq.weight                    -> blk.65.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.65.attention.wk.weight                    -> blk.65.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.65.attention.wv.weight                    -> blk.65.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.65.attention.wo.weight                    -> blk.65.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.65.feed_forward.w1.weight                 -> blk.65.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.65.feed_forward.w2.weight                 -> blk.65.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.65.feed_forward.w3.weight                 -> blk.65.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.65.attention_norm.weight                  -> blk.65.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.65.ffn_norm.weight                        -> blk.65.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.66.attention.wq.weight                    -> blk.66.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.66.attention.wk.weight                    -> blk.66.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.66.attention.wv.weight                    -> blk.66.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.66.attention.wo.weight                    -> blk.66.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.66.feed_forward.w1.weight                 -> blk.66.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.66.feed_forward.w2.weight                 -> blk.66.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.66.feed_forward.w3.weight                 -> blk.66.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.66.attention_norm.weight                  -> blk.66.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.66.ffn_norm.weight                        -> blk.66.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.67.attention.wq.weight                    -> blk.67.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.67.attention.wk.weight                    -> blk.67.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.67.attention.wv.weight                    -> blk.67.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.67.attention.wo.weight                    -> blk.67.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.67.feed_forward.w1.weight                 -> blk.67.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.67.feed_forward.w2.weight                 -> blk.67.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.67.feed_forward.w3.weight                 -> blk.67.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.67.attention_norm.weight                  -> blk.67.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.67.ffn_norm.weight                        -> blk.67.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.68.attention.wq.weight                    -> blk.68.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.68.attention.wk.weight                    -> blk.68.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.68.attention.wv.weight                    -> blk.68.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.68.attention.wo.weight                    -> blk.68.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.68.feed_forward.w1.weight                 -> blk.68.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.68.feed_forward.w2.weight                 -> blk.68.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.68.feed_forward.w3.weight                 -> blk.68.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.68.attention_norm.weight                  -> blk.68.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.68.ffn_norm.weight                        -> blk.68.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.69.attention.wq.weight                    -> blk.69.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.69.attention.wk.weight                    -> blk.69.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.69.attention.wv.weight                    -> blk.69.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.69.attention.wo.weight                    -> blk.69.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.69.feed_forward.w1.weight                 -> blk.69.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.69.feed_forward.w2.weight                 -> blk.69.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.69.feed_forward.w3.weight                 -> blk.69.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.69.attention_norm.weight                  -> blk.69.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.69.ffn_norm.weight                        -> blk.69.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.70.attention.wq.weight                    -> blk.70.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.70.attention.wk.weight                    -> blk.70.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.70.attention.wv.weight                    -> blk.70.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.70.attention.wo.weight                    -> blk.70.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.70.feed_forward.w1.weight                 -> blk.70.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.70.feed_forward.w2.weight                 -> blk.70.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.70.feed_forward.w3.weight                 -> blk.70.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.70.attention_norm.weight                  -> blk.70.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.70.ffn_norm.weight                        -> blk.70.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.71.attention.wq.weight                    -> blk.71.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.71.attention.wk.weight                    -> blk.71.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.71.attention.wv.weight                    -> blk.71.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.71.attention.wo.weight                    -> blk.71.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.71.feed_forward.w1.weight                 -> blk.71.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.71.feed_forward.w2.weight                 -> blk.71.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.71.feed_forward.w3.weight                 -> blk.71.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.71.attention_norm.weight                  -> blk.71.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.71.ffn_norm.weight                        -> blk.71.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.72.attention.wq.weight                    -> blk.72.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.72.attention.wk.weight                    -> blk.72.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.72.attention.wv.weight                    -> blk.72.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.72.attention.wo.weight                    -> blk.72.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.72.feed_forward.w1.weight                 -> blk.72.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.72.feed_forward.w2.weight                 -> blk.72.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.72.feed_forward.w3.weight                 -> blk.72.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.72.attention_norm.weight                  -> blk.72.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.72.ffn_norm.weight                        -> blk.72.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.73.attention.wq.weight                    -> blk.73.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.73.attention.wk.weight                    -> blk.73.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.73.attention.wv.weight                    -> blk.73.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.73.attention.wo.weight                    -> blk.73.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.73.feed_forward.w1.weight                 -> blk.73.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.73.feed_forward.w2.weight                 -> blk.73.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.73.feed_forward.w3.weight                 -> blk.73.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.73.attention_norm.weight                  -> blk.73.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.73.ffn_norm.weight                        -> blk.73.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.74.attention.wq.weight                    -> blk.74.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.74.attention.wk.weight                    -> blk.74.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.74.attention.wv.weight                    -> blk.74.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.74.attention.wo.weight                    -> blk.74.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.74.feed_forward.w1.weight                 -> blk.74.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.74.feed_forward.w2.weight                 -> blk.74.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.74.feed_forward.w3.weight                 -> blk.74.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.74.attention_norm.weight                  -> blk.74.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.74.ffn_norm.weight                        -> blk.74.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.75.attention.wq.weight                    -> blk.75.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.75.attention.wk.weight                    -> blk.75.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.75.attention.wv.weight                    -> blk.75.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.75.attention.wo.weight                    -> blk.75.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.75.feed_forward.w1.weight                 -> blk.75.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.75.feed_forward.w2.weight                 -> blk.75.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.75.feed_forward.w3.weight                 -> blk.75.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.75.attention_norm.weight                  -> blk.75.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.75.ffn_norm.weight                        -> blk.75.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.76.attention.wq.weight                    -> blk.76.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.76.attention.wk.weight                    -> blk.76.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.76.attention.wv.weight                    -> blk.76.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.76.attention.wo.weight                    -> blk.76.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.76.feed_forward.w1.weight                 -> blk.76.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.76.feed_forward.w2.weight                 -> blk.76.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.76.feed_forward.w3.weight                 -> blk.76.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.76.attention_norm.weight                  -> blk.76.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.76.ffn_norm.weight                        -> blk.76.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.77.attention.wq.weight                    -> blk.77.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.77.attention.wk.weight                    -> blk.77.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.77.attention.wv.weight                    -> blk.77.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.77.attention.wo.weight                    -> blk.77.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.77.feed_forward.w1.weight                 -> blk.77.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.77.feed_forward.w2.weight                 -> blk.77.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.77.feed_forward.w3.weight                 -> blk.77.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.77.attention_norm.weight                  -> blk.77.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.77.ffn_norm.weight                        -> blk.77.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.78.attention.wq.weight                    -> blk.78.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.78.attention.wk.weight                    -> blk.78.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.78.attention.wv.weight                    -> blk.78.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.78.attention.wo.weight                    -> blk.78.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.78.feed_forward.w1.weight                 -> blk.78.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.78.feed_forward.w2.weight                 -> blk.78.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.78.feed_forward.w3.weight                 -> blk.78.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.78.attention_norm.weight                  -> blk.78.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.78.ffn_norm.weight                        -> blk.78.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.79.attention.wq.weight                    -> blk.79.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.79.attention.wk.weight                    -> blk.79.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.79.attention.wv.weight                    -> blk.79.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.79.attention.wo.weight                    -> blk.79.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.79.feed_forward.w1.weight                 -> blk.79.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.79.feed_forward.w2.weight                 -> blk.79.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.79.feed_forward.w3.weight                 -> blk.79.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.79.attention_norm.weight                  -> blk.79.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.79.ffn_norm.weight                        -> blk.79.ffn_norm.weight                   | BF16   | [8192]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/70B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/723] Writing tensor token_embd.weight                      | size  32000 x   8192  | type F16  | T+   2\n",
      "[  2/723] Writing tensor output_norm.weight                     | size   8192           | type F32  | T+   2\n",
      "[  3/723] Writing tensor output.weight                          | size  32000 x   8192  | type F16  | T+   2\n",
      "[  4/723] Writing tensor blk.0.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   2\n",
      "[  5/723] Writing tensor blk.0.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   2\n",
      "[  6/723] Writing tensor blk.0.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   2\n",
      "[  7/723] Writing tensor blk.0.attn_output.weight               | size   8192 x   8192  | type F16  | T+   2\n",
      "[  8/723] Writing tensor blk.0.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   2\n",
      "[  9/723] Writing tensor blk.0.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   3\n",
      "[ 10/723] Writing tensor blk.0.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+   4\n",
      "[ 11/723] Writing tensor blk.0.attn_norm.weight                 | size   8192           | type F32  | T+   4\n",
      "[ 12/723] Writing tensor blk.0.ffn_norm.weight                  | size   8192           | type F32  | T+   4\n",
      "[ 13/723] Writing tensor blk.1.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   4\n",
      "[ 14/723] Writing tensor blk.1.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   4\n",
      "[ 15/723] Writing tensor blk.1.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   4\n",
      "[ 16/723] Writing tensor blk.1.attn_output.weight               | size   8192 x   8192  | type F16  | T+   4\n",
      "[ 17/723] Writing tensor blk.1.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   5\n",
      "[ 18/723] Writing tensor blk.1.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   5\n",
      "[ 19/723] Writing tensor blk.1.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+   6\n",
      "[ 20/723] Writing tensor blk.1.attn_norm.weight                 | size   8192           | type F32  | T+   6\n",
      "[ 21/723] Writing tensor blk.1.ffn_norm.weight                  | size   8192           | type F32  | T+   6\n",
      "[ 22/723] Writing tensor blk.2.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   6\n",
      "[ 23/723] Writing tensor blk.2.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   6\n",
      "[ 24/723] Writing tensor blk.2.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   6\n",
      "[ 25/723] Writing tensor blk.2.attn_output.weight               | size   8192 x   8192  | type F16  | T+   6\n",
      "[ 26/723] Writing tensor blk.2.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   7\n",
      "[ 27/723] Writing tensor blk.2.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   7\n",
      "[ 28/723] Writing tensor blk.2.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+   8\n",
      "[ 29/723] Writing tensor blk.2.attn_norm.weight                 | size   8192           | type F32  | T+   8\n",
      "[ 30/723] Writing tensor blk.2.ffn_norm.weight                  | size   8192           | type F32  | T+   8\n",
      "[ 31/723] Writing tensor blk.3.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   8\n",
      "[ 32/723] Writing tensor blk.3.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   8\n",
      "[ 33/723] Writing tensor blk.3.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   8\n",
      "[ 34/723] Writing tensor blk.3.attn_output.weight               | size   8192 x   8192  | type F16  | T+   8\n",
      "[ 35/723] Writing tensor blk.3.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   9\n",
      "[ 36/723] Writing tensor blk.3.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   9\n",
      "[ 37/723] Writing tensor blk.3.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  10\n",
      "[ 38/723] Writing tensor blk.3.attn_norm.weight                 | size   8192           | type F32  | T+  10\n",
      "[ 39/723] Writing tensor blk.3.ffn_norm.weight                  | size   8192           | type F32  | T+  10\n",
      "[ 40/723] Writing tensor blk.4.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  10\n",
      "[ 41/723] Writing tensor blk.4.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  10\n",
      "[ 42/723] Writing tensor blk.4.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  10\n",
      "[ 43/723] Writing tensor blk.4.attn_output.weight               | size   8192 x   8192  | type F16  | T+  10\n",
      "[ 44/723] Writing tensor blk.4.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  11\n",
      "[ 45/723] Writing tensor blk.4.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  12\n",
      "[ 46/723] Writing tensor blk.4.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  12\n",
      "[ 47/723] Writing tensor blk.4.attn_norm.weight                 | size   8192           | type F32  | T+  12\n",
      "[ 48/723] Writing tensor blk.4.ffn_norm.weight                  | size   8192           | type F32  | T+  12\n",
      "[ 49/723] Writing tensor blk.5.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  12\n",
      "[ 50/723] Writing tensor blk.5.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  12\n",
      "[ 51/723] Writing tensor blk.5.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  12\n",
      "[ 52/723] Writing tensor blk.5.attn_output.weight               | size   8192 x   8192  | type F16  | T+  12\n",
      "[ 53/723] Writing tensor blk.5.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  13\n",
      "[ 54/723] Writing tensor blk.5.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  14\n",
      "[ 55/723] Writing tensor blk.5.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  14\n",
      "[ 56/723] Writing tensor blk.5.attn_norm.weight                 | size   8192           | type F32  | T+  14\n",
      "[ 57/723] Writing tensor blk.5.ffn_norm.weight                  | size   8192           | type F32  | T+  14\n",
      "[ 58/723] Writing tensor blk.6.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  14\n",
      "[ 59/723] Writing tensor blk.6.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  14\n",
      "[ 60/723] Writing tensor blk.6.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  14\n",
      "[ 61/723] Writing tensor blk.6.attn_output.weight               | size   8192 x   8192  | type F16  | T+  14\n",
      "[ 62/723] Writing tensor blk.6.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  15\n",
      "[ 63/723] Writing tensor blk.6.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  16\n",
      "[ 64/723] Writing tensor blk.6.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  16\n",
      "[ 65/723] Writing tensor blk.6.attn_norm.weight                 | size   8192           | type F32  | T+  16\n",
      "[ 66/723] Writing tensor blk.6.ffn_norm.weight                  | size   8192           | type F32  | T+  16\n",
      "[ 67/723] Writing tensor blk.7.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  16\n",
      "[ 68/723] Writing tensor blk.7.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  16\n",
      "[ 69/723] Writing tensor blk.7.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  16\n",
      "[ 70/723] Writing tensor blk.7.attn_output.weight               | size   8192 x   8192  | type F16  | T+  16\n",
      "[ 71/723] Writing tensor blk.7.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  18\n",
      "[ 72/723] Writing tensor blk.7.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  18\n",
      "[ 73/723] Writing tensor blk.7.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  18\n",
      "[ 74/723] Writing tensor blk.7.attn_norm.weight                 | size   8192           | type F32  | T+  18\n",
      "[ 75/723] Writing tensor blk.7.ffn_norm.weight                  | size   8192           | type F32  | T+  18\n",
      "[ 76/723] Writing tensor blk.8.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  18\n",
      "[ 77/723] Writing tensor blk.8.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  18\n",
      "[ 78/723] Writing tensor blk.8.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  18\n",
      "[ 79/723] Writing tensor blk.8.attn_output.weight               | size   8192 x   8192  | type F16  | T+  18\n",
      "[ 80/723] Writing tensor blk.8.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  20\n",
      "[ 81/723] Writing tensor blk.8.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  20\n",
      "[ 82/723] Writing tensor blk.8.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  20\n",
      "[ 83/723] Writing tensor blk.8.attn_norm.weight                 | size   8192           | type F32  | T+  20\n",
      "[ 84/723] Writing tensor blk.8.ffn_norm.weight                  | size   8192           | type F32  | T+  20\n",
      "[ 85/723] Writing tensor blk.9.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  20\n",
      "[ 86/723] Writing tensor blk.9.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  20\n",
      "[ 87/723] Writing tensor blk.9.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  20\n",
      "[ 88/723] Writing tensor blk.9.attn_output.weight               | size   8192 x   8192  | type F16  | T+  20\n",
      "[ 89/723] Writing tensor blk.9.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  22\n",
      "[ 90/723] Writing tensor blk.9.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  22\n",
      "[ 91/723] Writing tensor blk.9.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  22\n",
      "[ 92/723] Writing tensor blk.9.attn_norm.weight                 | size   8192           | type F32  | T+  22\n",
      "[ 93/723] Writing tensor blk.9.ffn_norm.weight                  | size   8192           | type F32  | T+  22\n",
      "[ 94/723] Writing tensor blk.10.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  22\n",
      "[ 95/723] Writing tensor blk.10.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  22\n",
      "[ 96/723] Writing tensor blk.10.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  22\n",
      "[ 97/723] Writing tensor blk.10.attn_output.weight              | size   8192 x   8192  | type F16  | T+  22\n",
      "[ 98/723] Writing tensor blk.10.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  24\n",
      "[ 99/723] Writing tensor blk.10.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  24\n",
      "[100/723] Writing tensor blk.10.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  24\n",
      "[101/723] Writing tensor blk.10.attn_norm.weight                | size   8192           | type F32  | T+  24\n",
      "[102/723] Writing tensor blk.10.ffn_norm.weight                 | size   8192           | type F32  | T+  24\n",
      "[103/723] Writing tensor blk.11.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  24\n",
      "[104/723] Writing tensor blk.11.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  24\n",
      "[105/723] Writing tensor blk.11.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  24\n",
      "[106/723] Writing tensor blk.11.attn_output.weight              | size   8192 x   8192  | type F16  | T+  24\n",
      "[107/723] Writing tensor blk.11.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  26\n",
      "[108/723] Writing tensor blk.11.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  26\n",
      "[109/723] Writing tensor blk.11.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  26\n",
      "[110/723] Writing tensor blk.11.attn_norm.weight                | size   8192           | type F32  | T+  26\n",
      "[111/723] Writing tensor blk.11.ffn_norm.weight                 | size   8192           | type F32  | T+  26\n",
      "[112/723] Writing tensor blk.12.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  26\n",
      "[113/723] Writing tensor blk.12.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  26\n",
      "[114/723] Writing tensor blk.12.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  26\n",
      "[115/723] Writing tensor blk.12.attn_output.weight              | size   8192 x   8192  | type F16  | T+  26\n",
      "[116/723] Writing tensor blk.12.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  28\n",
      "[117/723] Writing tensor blk.12.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  28\n",
      "[118/723] Writing tensor blk.12.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  28\n",
      "[119/723] Writing tensor blk.12.attn_norm.weight                | size   8192           | type F32  | T+  28\n",
      "[120/723] Writing tensor blk.12.ffn_norm.weight                 | size   8192           | type F32  | T+  28\n",
      "[121/723] Writing tensor blk.13.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  28\n",
      "[122/723] Writing tensor blk.13.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  28\n",
      "[123/723] Writing tensor blk.13.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  28\n",
      "[124/723] Writing tensor blk.13.attn_output.weight              | size   8192 x   8192  | type F16  | T+  28\n",
      "[125/723] Writing tensor blk.13.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  30\n",
      "[126/723] Writing tensor blk.13.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  30\n",
      "[127/723] Writing tensor blk.13.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  31\n",
      "[128/723] Writing tensor blk.13.attn_norm.weight                | size   8192           | type F32  | T+  31\n",
      "[129/723] Writing tensor blk.13.ffn_norm.weight                 | size   8192           | type F32  | T+  31\n",
      "[130/723] Writing tensor blk.14.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  31\n",
      "[131/723] Writing tensor blk.14.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  31\n",
      "[132/723] Writing tensor blk.14.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  31\n",
      "[133/723] Writing tensor blk.14.attn_output.weight              | size   8192 x   8192  | type F16  | T+  31\n",
      "[134/723] Writing tensor blk.14.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  32\n",
      "[135/723] Writing tensor blk.14.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  33\n",
      "[136/723] Writing tensor blk.14.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  33\n",
      "[137/723] Writing tensor blk.14.attn_norm.weight                | size   8192           | type F32  | T+  33\n",
      "[138/723] Writing tensor blk.14.ffn_norm.weight                 | size   8192           | type F32  | T+  33\n",
      "[139/723] Writing tensor blk.15.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  33\n",
      "[140/723] Writing tensor blk.15.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  33\n",
      "[141/723] Writing tensor blk.15.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  33\n",
      "[142/723] Writing tensor blk.15.attn_output.weight              | size   8192 x   8192  | type F16  | T+  33\n",
      "[143/723] Writing tensor blk.15.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  35\n",
      "[144/723] Writing tensor blk.15.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  35\n",
      "[145/723] Writing tensor blk.15.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  35\n",
      "[146/723] Writing tensor blk.15.attn_norm.weight                | size   8192           | type F32  | T+  36\n",
      "[147/723] Writing tensor blk.15.ffn_norm.weight                 | size   8192           | type F32  | T+  36\n",
      "[148/723] Writing tensor blk.16.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  36\n",
      "[149/723] Writing tensor blk.16.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  36\n",
      "[150/723] Writing tensor blk.16.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  36\n",
      "[151/723] Writing tensor blk.16.attn_output.weight              | size   8192 x   8192  | type F16  | T+  36\n",
      "[152/723] Writing tensor blk.16.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  38\n",
      "[153/723] Writing tensor blk.16.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  38\n",
      "[154/723] Writing tensor blk.16.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  38\n",
      "[155/723] Writing tensor blk.16.attn_norm.weight                | size   8192           | type F32  | T+  39\n",
      "[156/723] Writing tensor blk.16.ffn_norm.weight                 | size   8192           | type F32  | T+  39\n",
      "[157/723] Writing tensor blk.17.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  39\n",
      "[158/723] Writing tensor blk.17.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  39\n",
      "[159/723] Writing tensor blk.17.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  39\n",
      "[160/723] Writing tensor blk.17.attn_output.weight              | size   8192 x   8192  | type F16  | T+  39\n",
      "[161/723] Writing tensor blk.17.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  40\n",
      "[162/723] Writing tensor blk.17.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  40\n",
      "[163/723] Writing tensor blk.17.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  41\n",
      "[164/723] Writing tensor blk.17.attn_norm.weight                | size   8192           | type F32  | T+  41\n",
      "[165/723] Writing tensor blk.17.ffn_norm.weight                 | size   8192           | type F32  | T+  41\n",
      "[166/723] Writing tensor blk.18.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  41\n",
      "[167/723] Writing tensor blk.18.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  41\n",
      "[168/723] Writing tensor blk.18.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  41\n",
      "[169/723] Writing tensor blk.18.attn_output.weight              | size   8192 x   8192  | type F16  | T+  41\n",
      "[170/723] Writing tensor blk.18.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  42\n",
      "[171/723] Writing tensor blk.18.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  43\n",
      "[172/723] Writing tensor blk.18.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  43\n",
      "[173/723] Writing tensor blk.18.attn_norm.weight                | size   8192           | type F32  | T+  43\n",
      "[174/723] Writing tensor blk.18.ffn_norm.weight                 | size   8192           | type F32  | T+  43\n",
      "[175/723] Writing tensor blk.19.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  43\n",
      "[176/723] Writing tensor blk.19.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  43\n",
      "[177/723] Writing tensor blk.19.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  43\n",
      "[178/723] Writing tensor blk.19.attn_output.weight              | size   8192 x   8192  | type F16  | T+  43\n",
      "[179/723] Writing tensor blk.19.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  45\n",
      "[180/723] Writing tensor blk.19.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  45\n",
      "[181/723] Writing tensor blk.19.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  45\n",
      "[182/723] Writing tensor blk.19.attn_norm.weight                | size   8192           | type F32  | T+  45\n",
      "[183/723] Writing tensor blk.19.ffn_norm.weight                 | size   8192           | type F32  | T+  45\n",
      "[184/723] Writing tensor blk.20.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  45\n",
      "[185/723] Writing tensor blk.20.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  45\n",
      "[186/723] Writing tensor blk.20.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  45\n",
      "[187/723] Writing tensor blk.20.attn_output.weight              | size   8192 x   8192  | type F16  | T+  45\n",
      "[188/723] Writing tensor blk.20.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  47\n",
      "[189/723] Writing tensor blk.20.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  47\n",
      "[190/723] Writing tensor blk.20.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  47\n",
      "[191/723] Writing tensor blk.20.attn_norm.weight                | size   8192           | type F32  | T+  47\n",
      "[192/723] Writing tensor blk.20.ffn_norm.weight                 | size   8192           | type F32  | T+  47\n",
      "[193/723] Writing tensor blk.21.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  47\n",
      "[194/723] Writing tensor blk.21.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  47\n",
      "[195/723] Writing tensor blk.21.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  47\n",
      "[196/723] Writing tensor blk.21.attn_output.weight              | size   8192 x   8192  | type F16  | T+  47\n",
      "[197/723] Writing tensor blk.21.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  49\n",
      "[198/723] Writing tensor blk.21.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  49\n",
      "[199/723] Writing tensor blk.21.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  50\n",
      "[200/723] Writing tensor blk.21.attn_norm.weight                | size   8192           | type F32  | T+  50\n",
      "[201/723] Writing tensor blk.21.ffn_norm.weight                 | size   8192           | type F32  | T+  50\n",
      "[202/723] Writing tensor blk.22.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  50\n",
      "[203/723] Writing tensor blk.22.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  50\n",
      "[204/723] Writing tensor blk.22.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  50\n",
      "[205/723] Writing tensor blk.22.attn_output.weight              | size   8192 x   8192  | type F16  | T+  50\n",
      "[206/723] Writing tensor blk.22.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  52\n",
      "[207/723] Writing tensor blk.22.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  52\n",
      "[208/723] Writing tensor blk.22.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  52\n",
      "[209/723] Writing tensor blk.22.attn_norm.weight                | size   8192           | type F32  | T+  52\n",
      "[210/723] Writing tensor blk.22.ffn_norm.weight                 | size   8192           | type F32  | T+  52\n",
      "[211/723] Writing tensor blk.23.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  52\n",
      "[212/723] Writing tensor blk.23.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  52\n",
      "[213/723] Writing tensor blk.23.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  52\n",
      "[214/723] Writing tensor blk.23.attn_output.weight              | size   8192 x   8192  | type F16  | T+  52\n",
      "[215/723] Writing tensor blk.23.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  54\n",
      "[216/723] Writing tensor blk.23.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  54\n",
      "[217/723] Writing tensor blk.23.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  55\n",
      "[218/723] Writing tensor blk.23.attn_norm.weight                | size   8192           | type F32  | T+  55\n",
      "[219/723] Writing tensor blk.23.ffn_norm.weight                 | size   8192           | type F32  | T+  55\n",
      "[220/723] Writing tensor blk.24.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  55\n",
      "[221/723] Writing tensor blk.24.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  55\n",
      "[222/723] Writing tensor blk.24.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  55\n",
      "[223/723] Writing tensor blk.24.attn_output.weight              | size   8192 x   8192  | type F16  | T+  55\n",
      "[224/723] Writing tensor blk.24.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  56\n",
      "[225/723] Writing tensor blk.24.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  57\n",
      "[226/723] Writing tensor blk.24.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  57\n",
      "[227/723] Writing tensor blk.24.attn_norm.weight                | size   8192           | type F32  | T+  57\n",
      "[228/723] Writing tensor blk.24.ffn_norm.weight                 | size   8192           | type F32  | T+  57\n",
      "[229/723] Writing tensor blk.25.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  57\n",
      "[230/723] Writing tensor blk.25.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  57\n",
      "[231/723] Writing tensor blk.25.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  57\n",
      "[232/723] Writing tensor blk.25.attn_output.weight              | size   8192 x   8192  | type F16  | T+  57\n",
      "[233/723] Writing tensor blk.25.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  59\n",
      "[234/723] Writing tensor blk.25.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  59\n",
      "[235/723] Writing tensor blk.25.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  59\n",
      "[236/723] Writing tensor blk.25.attn_norm.weight                | size   8192           | type F32  | T+  59\n",
      "[237/723] Writing tensor blk.25.ffn_norm.weight                 | size   8192           | type F32  | T+  59\n",
      "[238/723] Writing tensor blk.26.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  59\n",
      "[239/723] Writing tensor blk.26.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  59\n",
      "[240/723] Writing tensor blk.26.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  59\n",
      "[241/723] Writing tensor blk.26.attn_output.weight              | size   8192 x   8192  | type F16  | T+  59\n",
      "[242/723] Writing tensor blk.26.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  61\n",
      "[243/723] Writing tensor blk.26.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  61\n",
      "[244/723] Writing tensor blk.26.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  61\n",
      "[245/723] Writing tensor blk.26.attn_norm.weight                | size   8192           | type F32  | T+  61\n",
      "[246/723] Writing tensor blk.26.ffn_norm.weight                 | size   8192           | type F32  | T+  61\n",
      "[247/723] Writing tensor blk.27.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  61\n",
      "[248/723] Writing tensor blk.27.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  61\n",
      "[249/723] Writing tensor blk.27.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  61\n",
      "[250/723] Writing tensor blk.27.attn_output.weight              | size   8192 x   8192  | type F16  | T+  61\n",
      "[251/723] Writing tensor blk.27.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  63\n",
      "[252/723] Writing tensor blk.27.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  63\n",
      "[253/723] Writing tensor blk.27.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  63\n",
      "[254/723] Writing tensor blk.27.attn_norm.weight                | size   8192           | type F32  | T+  64\n",
      "[255/723] Writing tensor blk.27.ffn_norm.weight                 | size   8192           | type F32  | T+  64\n",
      "[256/723] Writing tensor blk.28.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  64\n",
      "[257/723] Writing tensor blk.28.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  64\n",
      "[258/723] Writing tensor blk.28.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  64\n",
      "[259/723] Writing tensor blk.28.attn_output.weight              | size   8192 x   8192  | type F16  | T+  64\n",
      "[260/723] Writing tensor blk.28.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  65\n",
      "[261/723] Writing tensor blk.28.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  66\n",
      "[262/723] Writing tensor blk.28.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  66\n",
      "[263/723] Writing tensor blk.28.attn_norm.weight                | size   8192           | type F32  | T+  66\n",
      "[264/723] Writing tensor blk.28.ffn_norm.weight                 | size   8192           | type F32  | T+  66\n",
      "[265/723] Writing tensor blk.29.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  66\n",
      "[266/723] Writing tensor blk.29.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  66\n",
      "[267/723] Writing tensor blk.29.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  66\n",
      "[268/723] Writing tensor blk.29.attn_output.weight              | size   8192 x   8192  | type F16  | T+  66\n",
      "[269/723] Writing tensor blk.29.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  68\n",
      "[270/723] Writing tensor blk.29.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  68\n",
      "[271/723] Writing tensor blk.29.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  68\n",
      "[272/723] Writing tensor blk.29.attn_norm.weight                | size   8192           | type F32  | T+  68\n",
      "[273/723] Writing tensor blk.29.ffn_norm.weight                 | size   8192           | type F32  | T+  68\n",
      "[274/723] Writing tensor blk.30.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  68\n",
      "[275/723] Writing tensor blk.30.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  68\n",
      "[276/723] Writing tensor blk.30.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  68\n",
      "[277/723] Writing tensor blk.30.attn_output.weight              | size   8192 x   8192  | type F16  | T+  68\n",
      "[278/723] Writing tensor blk.30.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  70\n",
      "[279/723] Writing tensor blk.30.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  70\n",
      "[280/723] Writing tensor blk.30.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  70\n",
      "[281/723] Writing tensor blk.30.attn_norm.weight                | size   8192           | type F32  | T+  70\n",
      "[282/723] Writing tensor blk.30.ffn_norm.weight                 | size   8192           | type F32  | T+  70\n",
      "[283/723] Writing tensor blk.31.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  70\n",
      "[284/723] Writing tensor blk.31.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  70\n",
      "[285/723] Writing tensor blk.31.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  70\n",
      "[286/723] Writing tensor blk.31.attn_output.weight              | size   8192 x   8192  | type F16  | T+  70\n",
      "[287/723] Writing tensor blk.31.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  72\n",
      "[288/723] Writing tensor blk.31.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  72\n",
      "[289/723] Writing tensor blk.31.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  72\n",
      "[290/723] Writing tensor blk.31.attn_norm.weight                | size   8192           | type F32  | T+  73\n",
      "[291/723] Writing tensor blk.31.ffn_norm.weight                 | size   8192           | type F32  | T+  73\n",
      "[292/723] Writing tensor blk.32.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  73\n",
      "[293/723] Writing tensor blk.32.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  73\n",
      "[294/723] Writing tensor blk.32.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  73\n",
      "[295/723] Writing tensor blk.32.attn_output.weight              | size   8192 x   8192  | type F16  | T+  73\n",
      "[296/723] Writing tensor blk.32.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  74\n",
      "[297/723] Writing tensor blk.32.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  74\n",
      "[298/723] Writing tensor blk.32.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  75\n",
      "[299/723] Writing tensor blk.32.attn_norm.weight                | size   8192           | type F32  | T+  75\n",
      "[300/723] Writing tensor blk.32.ffn_norm.weight                 | size   8192           | type F32  | T+  75\n",
      "[301/723] Writing tensor blk.33.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  75\n",
      "[302/723] Writing tensor blk.33.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  75\n",
      "[303/723] Writing tensor blk.33.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  75\n",
      "[304/723] Writing tensor blk.33.attn_output.weight              | size   8192 x   8192  | type F16  | T+  75\n",
      "[305/723] Writing tensor blk.33.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  77\n",
      "[306/723] Writing tensor blk.33.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  77\n",
      "[307/723] Writing tensor blk.33.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  77\n",
      "[308/723] Writing tensor blk.33.attn_norm.weight                | size   8192           | type F32  | T+  77\n",
      "[309/723] Writing tensor blk.33.ffn_norm.weight                 | size   8192           | type F32  | T+  77\n",
      "[310/723] Writing tensor blk.34.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  77\n",
      "[311/723] Writing tensor blk.34.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  77\n",
      "[312/723] Writing tensor blk.34.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  77\n",
      "[313/723] Writing tensor blk.34.attn_output.weight              | size   8192 x   8192  | type F16  | T+  77\n",
      "[314/723] Writing tensor blk.34.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  79\n",
      "[315/723] Writing tensor blk.34.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  79\n",
      "[316/723] Writing tensor blk.34.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  79\n",
      "[317/723] Writing tensor blk.34.attn_norm.weight                | size   8192           | type F32  | T+  79\n",
      "[318/723] Writing tensor blk.34.ffn_norm.weight                 | size   8192           | type F32  | T+  79\n",
      "[319/723] Writing tensor blk.35.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  79\n",
      "[320/723] Writing tensor blk.35.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  80\n",
      "[321/723] Writing tensor blk.35.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  80\n",
      "[322/723] Writing tensor blk.35.attn_output.weight              | size   8192 x   8192  | type F16  | T+  80\n",
      "[323/723] Writing tensor blk.35.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  81\n",
      "[324/723] Writing tensor blk.35.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  81\n",
      "[325/723] Writing tensor blk.35.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  82\n",
      "[326/723] Writing tensor blk.35.attn_norm.weight                | size   8192           | type F32  | T+  82\n",
      "[327/723] Writing tensor blk.35.ffn_norm.weight                 | size   8192           | type F32  | T+  82\n",
      "[328/723] Writing tensor blk.36.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  82\n",
      "[329/723] Writing tensor blk.36.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  82\n",
      "[330/723] Writing tensor blk.36.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  82\n",
      "[331/723] Writing tensor blk.36.attn_output.weight              | size   8192 x   8192  | type F16  | T+  82\n",
      "[332/723] Writing tensor blk.36.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  83\n",
      "[333/723] Writing tensor blk.36.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  84\n",
      "[334/723] Writing tensor blk.36.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  84\n",
      "[335/723] Writing tensor blk.36.attn_norm.weight                | size   8192           | type F32  | T+  84\n",
      "[336/723] Writing tensor blk.36.ffn_norm.weight                 | size   8192           | type F32  | T+  84\n",
      "[337/723] Writing tensor blk.37.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  84\n",
      "[338/723] Writing tensor blk.37.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  85\n",
      "[339/723] Writing tensor blk.37.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  85\n",
      "[340/723] Writing tensor blk.37.attn_output.weight              | size   8192 x   8192  | type F16  | T+  85\n",
      "[341/723] Writing tensor blk.37.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  86\n",
      "[342/723] Writing tensor blk.37.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  86\n",
      "[343/723] Writing tensor blk.37.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  86\n",
      "[344/723] Writing tensor blk.37.attn_norm.weight                | size   8192           | type F32  | T+  87\n",
      "[345/723] Writing tensor blk.37.ffn_norm.weight                 | size   8192           | type F32  | T+  87\n",
      "[346/723] Writing tensor blk.38.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  87\n",
      "[347/723] Writing tensor blk.38.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  87\n",
      "[348/723] Writing tensor blk.38.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  87\n",
      "[349/723] Writing tensor blk.38.attn_output.weight              | size   8192 x   8192  | type F16  | T+  87\n",
      "[350/723] Writing tensor blk.38.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  88\n",
      "[351/723] Writing tensor blk.38.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  88\n",
      "[352/723] Writing tensor blk.38.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  89\n",
      "[353/723] Writing tensor blk.38.attn_norm.weight                | size   8192           | type F32  | T+  89\n",
      "[354/723] Writing tensor blk.38.ffn_norm.weight                 | size   8192           | type F32  | T+  89\n",
      "[355/723] Writing tensor blk.39.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  89\n",
      "[356/723] Writing tensor blk.39.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  89\n",
      "[357/723] Writing tensor blk.39.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  89\n",
      "[358/723] Writing tensor blk.39.attn_output.weight              | size   8192 x   8192  | type F16  | T+  89\n",
      "[359/723] Writing tensor blk.39.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  90\n",
      "[360/723] Writing tensor blk.39.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  91\n",
      "[361/723] Writing tensor blk.39.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  91\n",
      "[362/723] Writing tensor blk.39.attn_norm.weight                | size   8192           | type F32  | T+  91\n",
      "[363/723] Writing tensor blk.39.ffn_norm.weight                 | size   8192           | type F32  | T+  91\n",
      "[364/723] Writing tensor blk.40.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  91\n",
      "[365/723] Writing tensor blk.40.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  91\n",
      "[366/723] Writing tensor blk.40.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  91\n",
      "[367/723] Writing tensor blk.40.attn_output.weight              | size   8192 x   8192  | type F16  | T+  91\n",
      "[368/723] Writing tensor blk.40.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  93\n",
      "[369/723] Writing tensor blk.40.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  93\n",
      "[370/723] Writing tensor blk.40.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  93\n",
      "[371/723] Writing tensor blk.40.attn_norm.weight                | size   8192           | type F32  | T+  93\n",
      "[372/723] Writing tensor blk.40.ffn_norm.weight                 | size   8192           | type F32  | T+  93\n",
      "[373/723] Writing tensor blk.41.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  93\n",
      "[374/723] Writing tensor blk.41.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  93\n",
      "[375/723] Writing tensor blk.41.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  93\n",
      "[376/723] Writing tensor blk.41.attn_output.weight              | size   8192 x   8192  | type F16  | T+  93\n",
      "[377/723] Writing tensor blk.41.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  95\n",
      "[378/723] Writing tensor blk.41.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  95\n",
      "[379/723] Writing tensor blk.41.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  95\n",
      "[380/723] Writing tensor blk.41.attn_norm.weight                | size   8192           | type F32  | T+  95\n",
      "[381/723] Writing tensor blk.41.ffn_norm.weight                 | size   8192           | type F32  | T+  95\n",
      "[382/723] Writing tensor blk.42.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  95\n",
      "[383/723] Writing tensor blk.42.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  95\n",
      "[384/723] Writing tensor blk.42.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  95\n",
      "[385/723] Writing tensor blk.42.attn_output.weight              | size   8192 x   8192  | type F16  | T+  95\n",
      "[386/723] Writing tensor blk.42.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  97\n",
      "[387/723] Writing tensor blk.42.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  97\n",
      "[388/723] Writing tensor blk.42.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  97\n",
      "[389/723] Writing tensor blk.42.attn_norm.weight                | size   8192           | type F32  | T+  98\n",
      "[390/723] Writing tensor blk.42.ffn_norm.weight                 | size   8192           | type F32  | T+  98\n",
      "[391/723] Writing tensor blk.43.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  98\n",
      "[392/723] Writing tensor blk.43.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  98\n",
      "[393/723] Writing tensor blk.43.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  98\n",
      "[394/723] Writing tensor blk.43.attn_output.weight              | size   8192 x   8192  | type F16  | T+  98\n",
      "[395/723] Writing tensor blk.43.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  99\n",
      "[396/723] Writing tensor blk.43.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 100\n",
      "[397/723] Writing tensor blk.43.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 100\n",
      "[398/723] Writing tensor blk.43.attn_norm.weight                | size   8192           | type F32  | T+ 100\n",
      "[399/723] Writing tensor blk.43.ffn_norm.weight                 | size   8192           | type F32  | T+ 100\n",
      "[400/723] Writing tensor blk.44.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 100\n",
      "[401/723] Writing tensor blk.44.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 100\n",
      "[402/723] Writing tensor blk.44.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 100\n",
      "[403/723] Writing tensor blk.44.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 100\n",
      "[404/723] Writing tensor blk.44.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 102\n",
      "[405/723] Writing tensor blk.44.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 102\n",
      "[406/723] Writing tensor blk.44.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 102\n",
      "[407/723] Writing tensor blk.44.attn_norm.weight                | size   8192           | type F32  | T+ 102\n",
      "[408/723] Writing tensor blk.44.ffn_norm.weight                 | size   8192           | type F32  | T+ 102\n",
      "[409/723] Writing tensor blk.45.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 102\n",
      "[410/723] Writing tensor blk.45.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 102\n",
      "[411/723] Writing tensor blk.45.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 102\n",
      "[412/723] Writing tensor blk.45.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 102\n",
      "[413/723] Writing tensor blk.45.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 104\n",
      "[414/723] Writing tensor blk.45.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 104\n",
      "[415/723] Writing tensor blk.45.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 104\n",
      "[416/723] Writing tensor blk.45.attn_norm.weight                | size   8192           | type F32  | T+ 104\n",
      "[417/723] Writing tensor blk.45.ffn_norm.weight                 | size   8192           | type F32  | T+ 104\n",
      "[418/723] Writing tensor blk.46.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 104\n",
      "[419/723] Writing tensor blk.46.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 105\n",
      "[420/723] Writing tensor blk.46.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 105\n",
      "[421/723] Writing tensor blk.46.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 105\n",
      "[422/723] Writing tensor blk.46.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 106\n",
      "[423/723] Writing tensor blk.46.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 106\n",
      "[424/723] Writing tensor blk.46.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 107\n",
      "[425/723] Writing tensor blk.46.attn_norm.weight                | size   8192           | type F32  | T+ 107\n",
      "[426/723] Writing tensor blk.46.ffn_norm.weight                 | size   8192           | type F32  | T+ 107\n",
      "[427/723] Writing tensor blk.47.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 107\n",
      "[428/723] Writing tensor blk.47.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 107\n",
      "[429/723] Writing tensor blk.47.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 107\n",
      "[430/723] Writing tensor blk.47.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 107\n",
      "[431/723] Writing tensor blk.47.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 108\n",
      "[432/723] Writing tensor blk.47.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 109\n",
      "[433/723] Writing tensor blk.47.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 109\n",
      "[434/723] Writing tensor blk.47.attn_norm.weight                | size   8192           | type F32  | T+ 109\n",
      "[435/723] Writing tensor blk.47.ffn_norm.weight                 | size   8192           | type F32  | T+ 109\n",
      "[436/723] Writing tensor blk.48.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 109\n",
      "[437/723] Writing tensor blk.48.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 109\n",
      "[438/723] Writing tensor blk.48.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 109\n",
      "[439/723] Writing tensor blk.48.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 109\n",
      "[440/723] Writing tensor blk.48.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 111\n",
      "[441/723] Writing tensor blk.48.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 111\n",
      "[442/723] Writing tensor blk.48.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 111\n",
      "[443/723] Writing tensor blk.48.attn_norm.weight                | size   8192           | type F32  | T+ 111\n",
      "[444/723] Writing tensor blk.48.ffn_norm.weight                 | size   8192           | type F32  | T+ 111\n",
      "[445/723] Writing tensor blk.49.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 111\n",
      "[446/723] Writing tensor blk.49.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 111\n",
      "[447/723] Writing tensor blk.49.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 111\n",
      "[448/723] Writing tensor blk.49.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 111\n",
      "[449/723] Writing tensor blk.49.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 113\n",
      "[450/723] Writing tensor blk.49.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 113\n",
      "[451/723] Writing tensor blk.49.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 113\n",
      "[452/723] Writing tensor blk.49.attn_norm.weight                | size   8192           | type F32  | T+ 114\n",
      "[453/723] Writing tensor blk.49.ffn_norm.weight                 | size   8192           | type F32  | T+ 114\n",
      "[454/723] Writing tensor blk.50.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 114\n",
      "[455/723] Writing tensor blk.50.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 114\n",
      "[456/723] Writing tensor blk.50.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 114\n",
      "[457/723] Writing tensor blk.50.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 114\n",
      "[458/723] Writing tensor blk.50.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 115\n",
      "[459/723] Writing tensor blk.50.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 115\n",
      "[460/723] Writing tensor blk.50.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 116\n",
      "[461/723] Writing tensor blk.50.attn_norm.weight                | size   8192           | type F32  | T+ 116\n",
      "[462/723] Writing tensor blk.50.ffn_norm.weight                 | size   8192           | type F32  | T+ 116\n",
      "[463/723] Writing tensor blk.51.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 116\n",
      "[464/723] Writing tensor blk.51.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 116\n",
      "[465/723] Writing tensor blk.51.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 116\n",
      "[466/723] Writing tensor blk.51.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 116\n",
      "[467/723] Writing tensor blk.51.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 118\n",
      "[468/723] Writing tensor blk.51.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 118\n",
      "[469/723] Writing tensor blk.51.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 119\n",
      "[470/723] Writing tensor blk.51.attn_norm.weight                | size   8192           | type F32  | T+ 119\n",
      "[471/723] Writing tensor blk.51.ffn_norm.weight                 | size   8192           | type F32  | T+ 119\n",
      "[472/723] Writing tensor blk.52.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 119\n",
      "[473/723] Writing tensor blk.52.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 119\n",
      "[474/723] Writing tensor blk.52.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 119\n",
      "[475/723] Writing tensor blk.52.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 119\n",
      "[476/723] Writing tensor blk.52.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 121\n",
      "[477/723] Writing tensor blk.52.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 121\n",
      "[478/723] Writing tensor blk.52.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 122\n",
      "[479/723] Writing tensor blk.52.attn_norm.weight                | size   8192           | type F32  | T+ 122\n",
      "[480/723] Writing tensor blk.52.ffn_norm.weight                 | size   8192           | type F32  | T+ 122\n",
      "[481/723] Writing tensor blk.53.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 122\n",
      "[482/723] Writing tensor blk.53.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 122\n",
      "[483/723] Writing tensor blk.53.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 122\n",
      "[484/723] Writing tensor blk.53.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 122\n",
      "[485/723] Writing tensor blk.53.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 123\n",
      "[486/723] Writing tensor blk.53.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 124\n",
      "[487/723] Writing tensor blk.53.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 124\n",
      "[488/723] Writing tensor blk.53.attn_norm.weight                | size   8192           | type F32  | T+ 124\n",
      "[489/723] Writing tensor blk.53.ffn_norm.weight                 | size   8192           | type F32  | T+ 124\n",
      "[490/723] Writing tensor blk.54.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 124\n",
      "[491/723] Writing tensor blk.54.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 124\n",
      "[492/723] Writing tensor blk.54.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 124\n",
      "[493/723] Writing tensor blk.54.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 124\n",
      "[494/723] Writing tensor blk.54.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 126\n",
      "[495/723] Writing tensor blk.54.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 126\n",
      "[496/723] Writing tensor blk.54.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 126\n",
      "[497/723] Writing tensor blk.54.attn_norm.weight                | size   8192           | type F32  | T+ 126\n",
      "[498/723] Writing tensor blk.54.ffn_norm.weight                 | size   8192           | type F32  | T+ 126\n",
      "[499/723] Writing tensor blk.55.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 126\n",
      "[500/723] Writing tensor blk.55.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 126\n",
      "[501/723] Writing tensor blk.55.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 126\n",
      "[502/723] Writing tensor blk.55.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 126\n",
      "[503/723] Writing tensor blk.55.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 128\n",
      "[504/723] Writing tensor blk.55.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 128\n",
      "[505/723] Writing tensor blk.55.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 128\n",
      "[506/723] Writing tensor blk.55.attn_norm.weight                | size   8192           | type F32  | T+ 128\n",
      "[507/723] Writing tensor blk.55.ffn_norm.weight                 | size   8192           | type F32  | T+ 128\n",
      "[508/723] Writing tensor blk.56.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 128\n",
      "[509/723] Writing tensor blk.56.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 129\n",
      "[510/723] Writing tensor blk.56.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 129\n",
      "[511/723] Writing tensor blk.56.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 129\n",
      "[512/723] Writing tensor blk.56.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 130\n",
      "[513/723] Writing tensor blk.56.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 131\n",
      "[514/723] Writing tensor blk.56.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 131\n",
      "[515/723] Writing tensor blk.56.attn_norm.weight                | size   8192           | type F32  | T+ 131\n",
      "[516/723] Writing tensor blk.56.ffn_norm.weight                 | size   8192           | type F32  | T+ 131\n",
      "[517/723] Writing tensor blk.57.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 131\n",
      "[518/723] Writing tensor blk.57.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 131\n",
      "[519/723] Writing tensor blk.57.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 131\n",
      "[520/723] Writing tensor blk.57.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 131\n",
      "[521/723] Writing tensor blk.57.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 133\n",
      "[522/723] Writing tensor blk.57.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 133\n",
      "[523/723] Writing tensor blk.57.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 133\n",
      "[524/723] Writing tensor blk.57.attn_norm.weight                | size   8192           | type F32  | T+ 133\n",
      "[525/723] Writing tensor blk.57.ffn_norm.weight                 | size   8192           | type F32  | T+ 133\n",
      "[526/723] Writing tensor blk.58.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 133\n",
      "[527/723] Writing tensor blk.58.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 133\n",
      "[528/723] Writing tensor blk.58.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 133\n",
      "[529/723] Writing tensor blk.58.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 133\n",
      "[530/723] Writing tensor blk.58.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 135\n",
      "[531/723] Writing tensor blk.58.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 135\n",
      "[532/723] Writing tensor blk.58.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 136\n",
      "[533/723] Writing tensor blk.58.attn_norm.weight                | size   8192           | type F32  | T+ 136\n",
      "[534/723] Writing tensor blk.58.ffn_norm.weight                 | size   8192           | type F32  | T+ 136\n",
      "[535/723] Writing tensor blk.59.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 136\n",
      "[536/723] Writing tensor blk.59.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 136\n",
      "[537/723] Writing tensor blk.59.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 136\n",
      "[538/723] Writing tensor blk.59.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 136\n",
      "[539/723] Writing tensor blk.59.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 137\n",
      "[540/723] Writing tensor blk.59.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 138\n",
      "[541/723] Writing tensor blk.59.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 138\n",
      "[542/723] Writing tensor blk.59.attn_norm.weight                | size   8192           | type F32  | T+ 138\n",
      "[543/723] Writing tensor blk.59.ffn_norm.weight                 | size   8192           | type F32  | T+ 138\n",
      "[544/723] Writing tensor blk.60.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 138\n",
      "[545/723] Writing tensor blk.60.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 138\n",
      "[546/723] Writing tensor blk.60.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 138\n",
      "[547/723] Writing tensor blk.60.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 138\n",
      "[548/723] Writing tensor blk.60.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 140\n",
      "[549/723] Writing tensor blk.60.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 140\n",
      "[550/723] Writing tensor blk.60.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 140\n",
      "[551/723] Writing tensor blk.60.attn_norm.weight                | size   8192           | type F32  | T+ 140\n",
      "[552/723] Writing tensor blk.60.ffn_norm.weight                 | size   8192           | type F32  | T+ 140\n",
      "[553/723] Writing tensor blk.61.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 140\n",
      "[554/723] Writing tensor blk.61.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 140\n",
      "[555/723] Writing tensor blk.61.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 140\n",
      "[556/723] Writing tensor blk.61.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 140\n",
      "[557/723] Writing tensor blk.61.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 142\n",
      "[558/723] Writing tensor blk.61.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 142\n",
      "[559/723] Writing tensor blk.61.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 143\n",
      "[560/723] Writing tensor blk.61.attn_norm.weight                | size   8192           | type F32  | T+ 143\n",
      "[561/723] Writing tensor blk.61.ffn_norm.weight                 | size   8192           | type F32  | T+ 143\n",
      "[562/723] Writing tensor blk.62.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 143\n",
      "[563/723] Writing tensor blk.62.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 143\n",
      "[564/723] Writing tensor blk.62.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 143\n",
      "[565/723] Writing tensor blk.62.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 143\n",
      "[566/723] Writing tensor blk.62.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 144\n",
      "[567/723] Writing tensor blk.62.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 145\n",
      "[568/723] Writing tensor blk.62.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 145\n",
      "[569/723] Writing tensor blk.62.attn_norm.weight                | size   8192           | type F32  | T+ 145\n",
      "[570/723] Writing tensor blk.62.ffn_norm.weight                 | size   8192           | type F32  | T+ 145\n",
      "[571/723] Writing tensor blk.63.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 145\n",
      "[572/723] Writing tensor blk.63.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 145\n",
      "[573/723] Writing tensor blk.63.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 145\n",
      "[574/723] Writing tensor blk.63.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 145\n",
      "[575/723] Writing tensor blk.63.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 147\n",
      "[576/723] Writing tensor blk.63.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 147\n",
      "[577/723] Writing tensor blk.63.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 147\n",
      "[578/723] Writing tensor blk.63.attn_norm.weight                | size   8192           | type F32  | T+ 147\n",
      "[579/723] Writing tensor blk.63.ffn_norm.weight                 | size   8192           | type F32  | T+ 147\n",
      "[580/723] Writing tensor blk.64.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 147\n",
      "[581/723] Writing tensor blk.64.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 147\n",
      "[582/723] Writing tensor blk.64.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 147\n",
      "[583/723] Writing tensor blk.64.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 147\n",
      "[584/723] Writing tensor blk.64.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 149\n",
      "[585/723] Writing tensor blk.64.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 149\n",
      "[586/723] Writing tensor blk.64.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 149\n",
      "[587/723] Writing tensor blk.64.attn_norm.weight                | size   8192           | type F32  | T+ 149\n",
      "[588/723] Writing tensor blk.64.ffn_norm.weight                 | size   8192           | type F32  | T+ 149\n",
      "[589/723] Writing tensor blk.65.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 149\n",
      "[590/723] Writing tensor blk.65.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 149\n",
      "[591/723] Writing tensor blk.65.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 149\n",
      "[592/723] Writing tensor blk.65.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 149\n",
      "[593/723] Writing tensor blk.65.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 151\n",
      "[594/723] Writing tensor blk.65.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 151\n",
      "[595/723] Writing tensor blk.65.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 151\n",
      "[596/723] Writing tensor blk.65.attn_norm.weight                | size   8192           | type F32  | T+ 152\n",
      "[597/723] Writing tensor blk.65.ffn_norm.weight                 | size   8192           | type F32  | T+ 152\n",
      "[598/723] Writing tensor blk.66.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 152\n",
      "[599/723] Writing tensor blk.66.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 152\n",
      "[600/723] Writing tensor blk.66.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 152\n",
      "[601/723] Writing tensor blk.66.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 152\n",
      "[602/723] Writing tensor blk.66.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 153\n",
      "[603/723] Writing tensor blk.66.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 154\n",
      "[604/723] Writing tensor blk.66.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 154\n",
      "[605/723] Writing tensor blk.66.attn_norm.weight                | size   8192           | type F32  | T+ 154\n",
      "[606/723] Writing tensor blk.66.ffn_norm.weight                 | size   8192           | type F32  | T+ 154\n",
      "[607/723] Writing tensor blk.67.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 154\n",
      "[608/723] Writing tensor blk.67.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 154\n",
      "[609/723] Writing tensor blk.67.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 154\n",
      "[610/723] Writing tensor blk.67.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 154\n",
      "[611/723] Writing tensor blk.67.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 156\n",
      "[612/723] Writing tensor blk.67.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 156\n",
      "[613/723] Writing tensor blk.67.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 156\n",
      "[614/723] Writing tensor blk.67.attn_norm.weight                | size   8192           | type F32  | T+ 156\n",
      "[615/723] Writing tensor blk.67.ffn_norm.weight                 | size   8192           | type F32  | T+ 156\n",
      "[616/723] Writing tensor blk.68.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 156\n",
      "[617/723] Writing tensor blk.68.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 156\n",
      "[618/723] Writing tensor blk.68.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 156\n",
      "[619/723] Writing tensor blk.68.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 156\n",
      "[620/723] Writing tensor blk.68.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 158\n",
      "[621/723] Writing tensor blk.68.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 158\n",
      "[622/723] Writing tensor blk.68.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 158\n",
      "[623/723] Writing tensor blk.68.attn_norm.weight                | size   8192           | type F32  | T+ 159\n",
      "[624/723] Writing tensor blk.68.ffn_norm.weight                 | size   8192           | type F32  | T+ 159\n",
      "[625/723] Writing tensor blk.69.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 159\n",
      "[626/723] Writing tensor blk.69.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 159\n",
      "[627/723] Writing tensor blk.69.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 159\n",
      "[628/723] Writing tensor blk.69.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 159\n",
      "[629/723] Writing tensor blk.69.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 160\n",
      "[630/723] Writing tensor blk.69.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 161\n",
      "[631/723] Writing tensor blk.69.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 161\n",
      "[632/723] Writing tensor blk.69.attn_norm.weight                | size   8192           | type F32  | T+ 161\n",
      "[633/723] Writing tensor blk.69.ffn_norm.weight                 | size   8192           | type F32  | T+ 161\n",
      "[634/723] Writing tensor blk.70.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 161\n",
      "[635/723] Writing tensor blk.70.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 161\n",
      "[636/723] Writing tensor blk.70.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 161\n",
      "[637/723] Writing tensor blk.70.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 161\n",
      "[638/723] Writing tensor blk.70.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 163\n",
      "[639/723] Writing tensor blk.70.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 163\n",
      "[640/723] Writing tensor blk.70.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 163\n",
      "[641/723] Writing tensor blk.70.attn_norm.weight                | size   8192           | type F32  | T+ 163\n",
      "[642/723] Writing tensor blk.70.ffn_norm.weight                 | size   8192           | type F32  | T+ 163\n",
      "[643/723] Writing tensor blk.71.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 163\n",
      "[644/723] Writing tensor blk.71.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 163\n",
      "[645/723] Writing tensor blk.71.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 163\n",
      "[646/723] Writing tensor blk.71.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 163\n",
      "[647/723] Writing tensor blk.71.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 165\n",
      "[648/723] Writing tensor blk.71.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 165\n",
      "[649/723] Writing tensor blk.71.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 165\n",
      "[650/723] Writing tensor blk.71.attn_norm.weight                | size   8192           | type F32  | T+ 165\n",
      "[651/723] Writing tensor blk.71.ffn_norm.weight                 | size   8192           | type F32  | T+ 165\n",
      "[652/723] Writing tensor blk.72.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 165\n",
      "[653/723] Writing tensor blk.72.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 165\n",
      "[654/723] Writing tensor blk.72.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 165\n",
      "[655/723] Writing tensor blk.72.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 165\n",
      "[656/723] Writing tensor blk.72.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 167\n",
      "[657/723] Writing tensor blk.72.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 167\n",
      "[658/723] Writing tensor blk.72.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 167\n",
      "[659/723] Writing tensor blk.72.attn_norm.weight                | size   8192           | type F32  | T+ 168\n",
      "[660/723] Writing tensor blk.72.ffn_norm.weight                 | size   8192           | type F32  | T+ 168\n",
      "[661/723] Writing tensor blk.73.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 168\n",
      "[662/723] Writing tensor blk.73.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 168\n",
      "[663/723] Writing tensor blk.73.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 168\n",
      "[664/723] Writing tensor blk.73.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 168\n",
      "[665/723] Writing tensor blk.73.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 169\n",
      "[666/723] Writing tensor blk.73.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 169\n",
      "[667/723] Writing tensor blk.73.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 170\n",
      "[668/723] Writing tensor blk.73.attn_norm.weight                | size   8192           | type F32  | T+ 170\n",
      "[669/723] Writing tensor blk.73.ffn_norm.weight                 | size   8192           | type F32  | T+ 170\n",
      "[670/723] Writing tensor blk.74.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 170\n",
      "[671/723] Writing tensor blk.74.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 170\n",
      "[672/723] Writing tensor blk.74.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 170\n",
      "[673/723] Writing tensor blk.74.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 170\n",
      "[674/723] Writing tensor blk.74.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 172\n",
      "[675/723] Writing tensor blk.74.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 172\n",
      "[676/723] Writing tensor blk.74.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 172\n",
      "[677/723] Writing tensor blk.74.attn_norm.weight                | size   8192           | type F32  | T+ 172\n",
      "[678/723] Writing tensor blk.74.ffn_norm.weight                 | size   8192           | type F32  | T+ 172\n",
      "[679/723] Writing tensor blk.75.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 172\n",
      "[680/723] Writing tensor blk.75.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 172\n",
      "[681/723] Writing tensor blk.75.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 172\n",
      "[682/723] Writing tensor blk.75.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 172\n",
      "[683/723] Writing tensor blk.75.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 174\n",
      "[684/723] Writing tensor blk.75.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 174\n",
      "[685/723] Writing tensor blk.75.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 174\n",
      "[686/723] Writing tensor blk.75.attn_norm.weight                | size   8192           | type F32  | T+ 174\n",
      "[687/723] Writing tensor blk.75.ffn_norm.weight                 | size   8192           | type F32  | T+ 174\n",
      "[688/723] Writing tensor blk.76.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 174\n",
      "[689/723] Writing tensor blk.76.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 174\n",
      "[690/723] Writing tensor blk.76.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 174\n",
      "[691/723] Writing tensor blk.76.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 174\n",
      "[692/723] Writing tensor blk.76.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 176\n",
      "[693/723] Writing tensor blk.76.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 176\n",
      "[694/723] Writing tensor blk.76.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 176\n",
      "[695/723] Writing tensor blk.76.attn_norm.weight                | size   8192           | type F32  | T+ 177\n",
      "[696/723] Writing tensor blk.76.ffn_norm.weight                 | size   8192           | type F32  | T+ 177\n",
      "[697/723] Writing tensor blk.77.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 177\n",
      "[698/723] Writing tensor blk.77.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 177\n",
      "[699/723] Writing tensor blk.77.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 177\n",
      "[700/723] Writing tensor blk.77.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 177\n",
      "[701/723] Writing tensor blk.77.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 178\n",
      "[702/723] Writing tensor blk.77.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 179\n",
      "[703/723] Writing tensor blk.77.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 179\n",
      "[704/723] Writing tensor blk.77.attn_norm.weight                | size   8192           | type F32  | T+ 179\n",
      "[705/723] Writing tensor blk.77.ffn_norm.weight                 | size   8192           | type F32  | T+ 179\n",
      "[706/723] Writing tensor blk.78.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 179\n",
      "[707/723] Writing tensor blk.78.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 179\n",
      "[708/723] Writing tensor blk.78.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 179\n",
      "[709/723] Writing tensor blk.78.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 179\n",
      "[710/723] Writing tensor blk.78.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 181\n",
      "[711/723] Writing tensor blk.78.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 181\n",
      "[712/723] Writing tensor blk.78.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 181\n",
      "[713/723] Writing tensor blk.78.attn_norm.weight                | size   8192           | type F32  | T+ 181\n",
      "[714/723] Writing tensor blk.78.ffn_norm.weight                 | size   8192           | type F32  | T+ 181\n",
      "[715/723] Writing tensor blk.79.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 181\n",
      "[716/723] Writing tensor blk.79.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 181\n",
      "[717/723] Writing tensor blk.79.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 181\n",
      "[718/723] Writing tensor blk.79.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 181\n",
      "[719/723] Writing tensor blk.79.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 183\n",
      "[720/723] Writing tensor blk.79.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 183\n",
      "[721/723] Writing tensor blk.79.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 183\n",
      "[722/723] Writing tensor blk.79.attn_norm.weight                | size   8192           | type F32  | T+ 183\n",
      "[723/723] Writing tensor blk.79.ffn_norm.weight                 | size   8192           | type F32  | T+ 183\n",
      "Wrote models/70B-v2/ggml-model-f16.gguf\n",
      "/bin/bash: line 1: ./quantize: No such file or directory\n",
      "/bin/bash: line 1: ./quantize: No such file or directory\n",
      "/bin/bash: line 1: ./quantize: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "# convert the models to ggml FP16 format\n",
    "!python3 convert.py models/7B-v2/\n",
    "!python3 convert.py models/13B-v2/\n",
    "!python3 convert.py models/70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "954d1eb9-d1d6-4525-8b0f-3b5809ad2d84",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I llama.cpp build info: \n",
      "I UNAME_S:   Linux\n",
      "I UNAME_P:   x86_64\n",
      "I UNAME_M:   x86_64\n",
      "I CFLAGS:    -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion \n",
      "I CXXFLAGS:  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi\n",
      "I NVCCFLAGS:  \n",
      "I LDFLAGS:    \n",
      "I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "I CXX:       g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "\n",
      "rm -vrf *.o tests/*.o *.so *.dll benchmark-matmult common/build-info.cpp *.dot *.gcno tests/*.gcno *.gcda tests/*.gcda *.gcov tests/*.gcov lcov-report gcovr-report main quantize quantize-stats perplexity embedding vdot q8dot train-text-from-scratch convert-llama2c-to-ggml simple batched batched-bench save-load-state server gguf llama-bench libllava.a llava-cli baby-llama beam-search speculative infill tokenize benchmark-matmult parallel finetune export-lora lookahead lookup tests/test-c.o metal tests/test-llama-grammar tests/test-grammar-parser tests/test-double-float tests/test-grad0 tests/test-opt tests/test-quantize-fns tests/test-quantize-perf tests/test-sampling tests/test-tokenizer-0-llama tests/test-tokenizer-0-falcon tests/test-tokenizer-1-llama tests/test-tokenizer-1-bpe tests/test-rope tests/test-backend-ops\n",
      "removed 'build-info.o'\n",
      "removed 'common.o'\n",
      "removed 'console.o'\n",
      "removed 'ggml-alloc.o'\n",
      "removed 'ggml-backend.o'\n",
      "removed 'ggml-cuda.o'\n",
      "removed 'ggml-quants.o'\n",
      "removed 'ggml.o'\n",
      "removed 'grammar-parser.o'\n",
      "removed 'llama.o'\n",
      "removed 'sampling.o'\n",
      "removed 'train.o'\n",
      "removed 'tests/test-c.o'\n",
      "removed 'benchmark-matmult'\n",
      "removed 'common/build-info.cpp'\n",
      "removed 'main'\n",
      "removed 'quantize'\n",
      "removed 'quantize-stats'\n",
      "removed 'perplexity'\n",
      "removed 'embedding'\n",
      "removed 'vdot'\n",
      "removed 'q8dot'\n",
      "removed 'train-text-from-scratch'\n",
      "removed 'convert-llama2c-to-ggml'\n",
      "removed 'simple'\n",
      "removed 'batched'\n",
      "removed 'batched-bench'\n",
      "removed 'save-load-state'\n",
      "removed 'server'\n",
      "removed 'gguf'\n",
      "removed 'llama-bench'\n",
      "removed 'libllava.a'\n",
      "removed 'llava-cli'\n",
      "removed 'baby-llama'\n",
      "removed 'beam-search'\n",
      "removed 'speculative'\n",
      "removed 'infill'\n",
      "removed 'tokenize'\n",
      "removed 'parallel'\n",
      "removed 'finetune'\n",
      "removed 'export-lora'\n",
      "removed 'lookahead'\n",
      "removed 'lookup'\n",
      "I llama.cpp build info: \n",
      "I UNAME_S:   Linux\n",
      "I UNAME_P:   x86_64\n",
      "I UNAME_M:   x86_64\n",
      "I CFLAGS:    -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion \n",
      "I CXXFLAGS:  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi\n",
      "I NVCCFLAGS: -use_fast_math --forward-unknown-to-host-compiler -arch=native -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \n",
      "I LDFLAGS:   -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "I CXX:       g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml.c -o ggml.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c llama.cpp -o llama.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/common.cpp -o common.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/sampling.cpp -o sampling.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/grammar-parser.cpp -o grammar-parser.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/console.cpp -o console.o\n",
      "nvcc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -use_fast_math --forward-unknown-to-host-compiler -arch=native -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128  -Wno-pedantic -Xcompiler \"-Wno-array-bounds -Wno-format-truncation -Wextra-semi\" -c ggml-cuda.cu -o ggml-cuda.o\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-alloc.c -o ggml-alloc.o\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-backend.c -o ggml-backend.o\n",
      "cc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion     -c ggml-quants.c -o ggml-quants.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/train.cpp -o train.o\n",
      "cc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion  -c tests/test-c.c -o tests/test-c.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/build-info.cpp -o build-info.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi pocs/vdot/vdot.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o vdot -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi pocs/vdot/q8dot.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o q8dot -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/gguf/gguf.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o gguf -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/benchmark/benchmark-matmult.cpp build-info.o ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o benchmark-matmult -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/export-lora/export-lora.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o export-lora -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/main/main.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o console.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o main -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/quantize/quantize.cpp build-info.o ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o quantize -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/quantize-stats/quantize-stats.cpp build-info.o ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o quantize-stats -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/perplexity/perplexity.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o perplexity -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/embedding/embedding.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o embedding -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/train-text-from-scratch/train-text-from-scratch.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o train-text-from-scratch -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/convert-llama2c-to-ggml/convert-llama2c-to-ggml.cpp ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o convert-llama2c-to-ggml -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/simple/simple.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o simple -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/batched/batched.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o batched -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/batched-bench/batched-bench.cpp build-info.o ggml.o llama.o common.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o batched-bench -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/save-load-state/save-load-state.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o save-load-state -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -Iexamples/server examples/server/server.cpp examples/llava/clip.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o server -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib   -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/llama-bench/llama-bench.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o llama-bench -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -static -fPIC -c examples/llava/llava.cpp -o libllava.a -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/llava/llava-cli.cpp examples/llava/clip.cpp examples/llava/llava.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o llava-cli -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib  -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/baby-llama/baby-llama.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o baby-llama -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/beam-search/beam-search.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o beam-search -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/speculative/speculative.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o speculative -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/infill/infill.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o console.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o infill -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/tokenize/tokenize.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o tokenize -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/parallel/parallel.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o parallel -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/finetune/finetune.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o finetune -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/lookahead/lookahead.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o lookahead -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/lookup/lookup.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o lookup -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "\n",
      "====  Run ./main -h for help.  ====\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# metal build\n",
    "!make clean && LLAMA_CUBLAS=1 make -j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c99bdabe-ce05-4e4a-bb7f-1ad00b66e57e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "main: build = 1671 (8fe03ff)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/7B-v2/ggml-model-f16.gguf' to './models/7B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llama_model_quantize_internal: meta size = 1714336 bytes\n",
      "[   1/ 291]                    token_embd.weight - [ 4096, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   250.00 MiB ->    70.31 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   2/ 291]                   output_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[   3/ 291]                        output.weight - [ 4096, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   250.00 MiB ->   102.54 MiB | hist: \n",
      "[   4/ 291]                  blk.0.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.034 0.008 0.012 0.019 0.031 0.050 0.084 0.149 0.256 0.150 0.084 0.051 0.031 0.019 0.012 0.010 \n",
      "[   5/ 291]                  blk.0.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.034 0.008 0.013 0.021 0.033 0.054 0.089 0.150 0.226 0.151 0.089 0.054 0.033 0.021 0.013 0.011 \n",
      "[   6/ 291]                  blk.0.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.036 0.053 0.074 0.096 0.117 0.129 0.117 0.096 0.074 0.053 0.036 0.024 0.020 \n",
      "[   7/ 291]             blk.0.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.035 0.011 0.017 0.028 0.044 0.068 0.100 0.135 0.155 0.135 0.100 0.068 0.044 0.028 0.017 0.014 \n",
      "[   8/ 291]                blk.0.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   9/ 291]                blk.0.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  10/ 291]                  blk.0.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  11/ 291]               blk.0.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  12/ 291]                blk.0.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  13/ 291]                  blk.1.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.013 0.022 0.034 0.052 0.074 0.098 0.121 0.132 0.121 0.098 0.074 0.052 0.034 0.022 0.018 \n",
      "[  14/ 291]                  blk.1.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.013 0.022 0.034 0.051 0.074 0.099 0.121 0.132 0.121 0.099 0.074 0.051 0.034 0.022 0.018 \n",
      "[  15/ 291]                  blk.1.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.014 0.023 0.035 0.052 0.073 0.097 0.119 0.130 0.119 0.097 0.074 0.052 0.035 0.023 0.019 \n",
      "[  16/ 291]             blk.1.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.035 0.012 0.020 0.031 0.047 0.070 0.098 0.129 0.146 0.129 0.099 0.070 0.047 0.031 0.020 0.016 \n",
      "[  17/ 291]                blk.1.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 291]                blk.1.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 291]                  blk.1.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 291]               blk.1.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  21/ 291]                blk.1.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  22/ 291]                  blk.2.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.096 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  23/ 291]                  blk.2.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.124 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[  24/ 291]                  blk.2.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.120 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  25/ 291]             blk.2.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  26/ 291]                blk.2.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 291]                blk.2.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 291]                  blk.2.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 291]               blk.2.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  30/ 291]                blk.2.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  31/ 291]                  blk.3.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  32/ 291]                  blk.3.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  33/ 291]                  blk.3.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  34/ 291]             blk.3.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  35/ 291]                blk.3.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 291]                blk.3.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  37/ 291]                  blk.3.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 291]               blk.3.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  39/ 291]                blk.3.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  40/ 291]                  blk.4.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[  41/ 291]                  blk.4.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  42/ 291]                  blk.4.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  43/ 291]             blk.4.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  44/ 291]                blk.4.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 291]                blk.4.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  46/ 291]                  blk.4.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  47/ 291]               blk.4.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  48/ 291]                blk.4.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  49/ 291]                  blk.5.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  50/ 291]                  blk.5.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 291]                  blk.5.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  52/ 291]             blk.5.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  53/ 291]                blk.5.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 291]                blk.5.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  55/ 291]                  blk.5.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  56/ 291]               blk.5.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  57/ 291]                blk.5.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  58/ 291]                  blk.6.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  59/ 291]                  blk.6.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  60/ 291]                  blk.6.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  61/ 291]             blk.6.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  62/ 291]                blk.6.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 291]                blk.6.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 291]                  blk.6.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 291]               blk.6.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  66/ 291]                blk.6.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  67/ 291]                  blk.7.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  68/ 291]                  blk.7.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  69/ 291]                  blk.7.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  70/ 291]             blk.7.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  71/ 291]                blk.7.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 291]                blk.7.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  73/ 291]                  blk.7.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 291]               blk.7.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  75/ 291]                blk.7.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  76/ 291]                  blk.8.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 291]                  blk.8.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 291]                  blk.8.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  79/ 291]             blk.8.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 291]                blk.8.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 291]                blk.8.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  82/ 291]                  blk.8.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  83/ 291]               blk.8.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  84/ 291]                blk.8.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  85/ 291]                  blk.9.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 291]                  blk.9.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  87/ 291]                  blk.9.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  88/ 291]             blk.9.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  89/ 291]                blk.9.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 291]                blk.9.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  91/ 291]                  blk.9.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  92/ 291]               blk.9.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  93/ 291]                blk.9.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  94/ 291]                 blk.10.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  95/ 291]                 blk.10.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  96/ 291]                 blk.10.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  97/ 291]            blk.10.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  98/ 291]               blk.10.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 291]               blk.10.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 100/ 291]                 blk.10.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 101/ 291]              blk.10.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 102/ 291]               blk.10.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 103/ 291]                 blk.11.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 291]                 blk.11.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 291]                 blk.11.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 106/ 291]            blk.11.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 107/ 291]               blk.11.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 291]               blk.11.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 109/ 291]                 blk.11.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 110/ 291]              blk.11.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 111/ 291]               blk.11.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 112/ 291]                 blk.12.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 113/ 291]                 blk.12.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 291]                 blk.12.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 291]            blk.12.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 116/ 291]               blk.12.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 291]               blk.12.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 118/ 291]                 blk.12.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 291]              blk.12.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 120/ 291]               blk.12.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 121/ 291]                 blk.13.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 291]                 blk.13.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 123/ 291]                 blk.13.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 291]            blk.13.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 125/ 291]               blk.13.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 291]               blk.13.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 127/ 291]                 blk.13.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 291]              blk.13.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 129/ 291]               blk.13.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 130/ 291]                 blk.14.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 131/ 291]                 blk.14.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 291]                 blk.14.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 291]            blk.14.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 134/ 291]               blk.14.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 291]               blk.14.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 136/ 291]                 blk.14.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 291]              blk.14.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 138/ 291]               blk.14.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 139/ 291]                 blk.15.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 291]                 blk.15.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 141/ 291]                 blk.15.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 291]            blk.15.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 143/ 291]               blk.15.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 291]               blk.15.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 145/ 291]                 blk.15.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 291]              blk.15.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 147/ 291]               blk.15.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 148/ 291]                 blk.16.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 291]                 blk.16.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 150/ 291]                 blk.16.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 291]            blk.16.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 152/ 291]               blk.16.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 291]               blk.16.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 154/ 291]                 blk.16.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 291]              blk.16.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 156/ 291]               blk.16.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 157/ 291]                 blk.17.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 291]                 blk.17.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 159/ 291]                 blk.17.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 291]            blk.17.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 161/ 291]               blk.17.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 291]               blk.17.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 163/ 291]                 blk.17.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 291]              blk.17.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 165/ 291]               blk.17.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 166/ 291]                 blk.18.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 167/ 291]                 blk.18.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 168/ 291]                 blk.18.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 291]            blk.18.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 170/ 291]               blk.18.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 291]               blk.18.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 172/ 291]                 blk.18.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 291]              blk.18.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 174/ 291]               blk.18.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 175/ 291]                 blk.19.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 176/ 291]                 blk.19.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 177/ 291]                 blk.19.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 291]            blk.19.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 179/ 291]               blk.19.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 291]               blk.19.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 291]                 blk.19.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 291]              blk.19.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 183/ 291]               blk.19.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 184/ 291]                 blk.20.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 291]                 blk.20.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 186/ 291]                 blk.20.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 291]            blk.20.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 188/ 291]               blk.20.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 291]               blk.20.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 291]                 blk.20.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 291]              blk.20.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 192/ 291]               blk.20.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 193/ 291]                 blk.21.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 194/ 291]                 blk.21.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 195/ 291]                 blk.21.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 291]            blk.21.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 291]               blk.21.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 291]               blk.21.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 291]                 blk.21.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 291]              blk.21.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 201/ 291]               blk.21.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 202/ 291]                 blk.22.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 203/ 291]                 blk.22.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 204/ 291]                 blk.22.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 205/ 291]            blk.22.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 291]               blk.22.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 291]               blk.22.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 291]                 blk.22.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 291]              blk.22.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 210/ 291]               blk.22.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 211/ 291]                 blk.23.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 212/ 291]                 blk.23.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 213/ 291]                 blk.23.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 214/ 291]            blk.23.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 215/ 291]               blk.23.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 291]               blk.23.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 291]                 blk.23.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 291]              blk.23.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 219/ 291]               blk.23.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 220/ 291]                 blk.24.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 221/ 291]                 blk.24.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 222/ 291]                 blk.24.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 291]            blk.24.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 224/ 291]               blk.24.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 291]               blk.24.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 291]                 blk.24.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 291]              blk.24.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 228/ 291]               blk.24.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 229/ 291]                 blk.25.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 291]                 blk.25.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 231/ 291]                 blk.25.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 291]            blk.25.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 233/ 291]               blk.25.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 291]               blk.25.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 291]                 blk.25.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 291]              blk.25.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 237/ 291]               blk.25.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 238/ 291]                 blk.26.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 239/ 291]                 blk.26.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 240/ 291]                 blk.26.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 241/ 291]            blk.26.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 242/ 291]               blk.26.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 291]               blk.26.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 291]                 blk.26.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 291]              blk.26.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 246/ 291]               blk.26.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 247/ 291]                 blk.27.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 248/ 291]                 blk.27.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 249/ 291]                 blk.27.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 291]            blk.27.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 291]               blk.27.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 291]               blk.27.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 291]                 blk.27.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 291]              blk.27.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 255/ 291]               blk.27.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 256/ 291]                 blk.28.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 257/ 291]                 blk.28.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 258/ 291]                 blk.28.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 291]            blk.28.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 260/ 291]               blk.28.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 291]               blk.28.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 262/ 291]                 blk.28.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 291]              blk.28.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 264/ 291]               blk.28.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 265/ 291]                 blk.29.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 291]                 blk.29.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 267/ 291]                 blk.29.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 291]            blk.29.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 269/ 291]               blk.29.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 291]               blk.29.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 271/ 291]                 blk.29.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 291]              blk.29.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 273/ 291]               blk.29.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 274/ 291]                 blk.30.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 275/ 291]                 blk.30.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 276/ 291]                 blk.30.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 277/ 291]            blk.30.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 291]               blk.30.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 291]               blk.30.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 280/ 291]                 blk.30.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 291]              blk.30.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 282/ 291]               blk.30.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 283/ 291]                 blk.31.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 284/ 291]                 blk.31.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 285/ 291]                 blk.31.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 286/ 291]            blk.31.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 287/ 291]               blk.31.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 291]               blk.31.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[ 289/ 291]                 blk.31.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 290/ 291]              blk.31.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 291/ 291]               blk.31.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "llama_model_quantize_internal: model size  = 12853.02 MB\n",
      "llama_model_quantize_internal: quant size  =  3647.87 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 17181.91 ms\n",
      "main:    total time = 17181.91 ms\n",
      "main: build = 1671 (8fe03ff)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/13B-v2/ggml-model-f16.gguf' to './models/13B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llama_model_quantize_internal: meta size = 1718656 bytes\n",
      "[   1/ 363]                    token_embd.weight - [ 5120, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   312.50 MiB ->    87.89 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   2/ 363]                   output_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[   3/ 363]                        output.weight - [ 5120, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   312.50 MiB ->   128.17 MiB | hist: \n",
      "[   4/ 363]                  blk.0.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.033 0.006 0.009 0.015 0.024 0.041 0.074 0.153 0.317 0.153 0.075 0.041 0.024 0.015 0.009 0.008 \n",
      "[   5/ 363]                  blk.0.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.033 0.006 0.010 0.015 0.025 0.043 0.078 0.158 0.293 0.158 0.078 0.043 0.025 0.015 0.010 0.008 \n",
      "[   6/ 363]                  blk.0.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.014 0.023 0.035 0.053 0.074 0.097 0.118 0.129 0.119 0.098 0.074 0.053 0.035 0.023 0.019 \n",
      "[   7/ 363]             blk.0.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.035 0.012 0.020 0.031 0.048 0.071 0.099 0.127 0.142 0.127 0.099 0.071 0.048 0.031 0.020 0.016 \n",
      "[   8/ 363]                blk.0.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[   9/ 363]                blk.0.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  10/ 363]                  blk.0.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  11/ 363]               blk.0.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  12/ 363]                blk.0.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  13/ 363]                  blk.1.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.072 0.098 0.124 0.139 0.124 0.098 0.072 0.050 0.033 0.021 0.017 \n",
      "[  14/ 363]                  blk.1.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.020 0.032 0.049 0.072 0.099 0.125 0.139 0.126 0.099 0.072 0.049 0.032 0.020 0.017 \n",
      "[  15/ 363]                  blk.1.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.075 0.097 0.116 0.124 0.116 0.097 0.075 0.054 0.037 0.024 0.020 \n",
      "[  16/ 363]             blk.1.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.021 0.033 0.051 0.073 0.099 0.123 0.134 0.123 0.099 0.073 0.051 0.034 0.021 0.018 \n",
      "[  17/ 363]                blk.1.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 363]                blk.1.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 363]                  blk.1.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 363]               blk.1.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  21/ 363]                blk.1.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  22/ 363]                  blk.2.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.123 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  23/ 363]                  blk.2.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.124 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[  24/ 363]                  blk.2.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  25/ 363]             blk.2.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.024 0.020 \n",
      "[  26/ 363]                blk.2.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 363]                blk.2.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 363]                  blk.2.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 363]               blk.2.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  30/ 363]                blk.2.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  31/ 363]                  blk.3.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.096 0.113 0.121 0.113 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[  32/ 363]                  blk.3.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[  33/ 363]                  blk.3.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  34/ 363]             blk.3.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  35/ 363]                blk.3.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 363]                blk.3.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  37/ 363]                  blk.3.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 363]               blk.3.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  39/ 363]                blk.3.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  40/ 363]                  blk.4.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  41/ 363]                  blk.4.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  42/ 363]                  blk.4.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[  43/ 363]             blk.4.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  44/ 363]                blk.4.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 363]                blk.4.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  46/ 363]                  blk.4.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  47/ 363]               blk.4.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  48/ 363]                blk.4.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  49/ 363]                  blk.5.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  50/ 363]                  blk.5.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 363]                  blk.5.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  52/ 363]             blk.5.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  53/ 363]                blk.5.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 363]                blk.5.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  55/ 363]                  blk.5.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  56/ 363]               blk.5.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  57/ 363]                blk.5.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  58/ 363]                  blk.6.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  59/ 363]                  blk.6.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  60/ 363]                  blk.6.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  61/ 363]             blk.6.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  62/ 363]                blk.6.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 363]                blk.6.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 363]                  blk.6.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 363]               blk.6.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  66/ 363]                blk.6.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  67/ 363]                  blk.7.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  68/ 363]                  blk.7.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  69/ 363]                  blk.7.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  70/ 363]             blk.7.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  71/ 363]                blk.7.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 363]                blk.7.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  73/ 363]                  blk.7.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 363]               blk.7.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  75/ 363]                blk.7.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  76/ 363]                  blk.8.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 363]                  blk.8.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 363]                  blk.8.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  79/ 363]             blk.8.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 363]                blk.8.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 363]                blk.8.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  82/ 363]                  blk.8.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  83/ 363]               blk.8.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  84/ 363]                blk.8.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  85/ 363]                  blk.9.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 363]                  blk.9.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  87/ 363]                  blk.9.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  88/ 363]             blk.9.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  89/ 363]                blk.9.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 363]                blk.9.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  91/ 363]                  blk.9.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  92/ 363]               blk.9.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  93/ 363]                blk.9.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  94/ 363]                 blk.10.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  95/ 363]                 blk.10.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  96/ 363]                 blk.10.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  97/ 363]            blk.10.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  98/ 363]               blk.10.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 363]               blk.10.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 100/ 363]                 blk.10.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 101/ 363]              blk.10.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 102/ 363]               blk.10.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 103/ 363]                 blk.11.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 363]                 blk.11.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 363]                 blk.11.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 106/ 363]            blk.11.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 107/ 363]               blk.11.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 363]               blk.11.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 109/ 363]                 blk.11.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 110/ 363]              blk.11.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 111/ 363]               blk.11.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 112/ 363]                 blk.12.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 113/ 363]                 blk.12.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 363]                 blk.12.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 363]            blk.12.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 116/ 363]               blk.12.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 363]               blk.12.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 118/ 363]                 blk.12.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 363]              blk.12.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 120/ 363]               blk.12.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 121/ 363]                 blk.13.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 363]                 blk.13.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 123/ 363]                 blk.13.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 363]            blk.13.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 125/ 363]               blk.13.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 363]               blk.13.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 127/ 363]                 blk.13.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 363]              blk.13.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 129/ 363]               blk.13.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 130/ 363]                 blk.14.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 131/ 363]                 blk.14.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 363]                 blk.14.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 363]            blk.14.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 134/ 363]               blk.14.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 363]               blk.14.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 136/ 363]                 blk.14.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 363]              blk.14.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 138/ 363]               blk.14.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 139/ 363]                 blk.15.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 363]                 blk.15.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 141/ 363]                 blk.15.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 363]            blk.15.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 143/ 363]               blk.15.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 363]               blk.15.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 145/ 363]                 blk.15.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 363]              blk.15.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 147/ 363]               blk.15.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 148/ 363]                 blk.16.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 363]                 blk.16.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 150/ 363]                 blk.16.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 363]            blk.16.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 152/ 363]               blk.16.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 363]               blk.16.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 154/ 363]                 blk.16.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 363]              blk.16.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 156/ 363]               blk.16.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 157/ 363]                 blk.17.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 363]                 blk.17.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 159/ 363]                 blk.17.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 363]            blk.17.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 161/ 363]               blk.17.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 363]               blk.17.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 163/ 363]                 blk.17.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 363]              blk.17.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 165/ 363]               blk.17.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 166/ 363]                 blk.18.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 167/ 363]                 blk.18.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 168/ 363]                 blk.18.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 363]            blk.18.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 170/ 363]               blk.18.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 363]               blk.18.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 172/ 363]                 blk.18.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 363]              blk.18.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 174/ 363]               blk.18.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 175/ 363]                 blk.19.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 176/ 363]                 blk.19.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 177/ 363]                 blk.19.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 363]            blk.19.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 179/ 363]               blk.19.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 363]               blk.19.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 363]                 blk.19.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 363]              blk.19.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 183/ 363]               blk.19.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 184/ 363]                 blk.20.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 363]                 blk.20.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 186/ 363]                 blk.20.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 363]            blk.20.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 188/ 363]               blk.20.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 363]               blk.20.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 363]                 blk.20.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 363]              blk.20.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 192/ 363]               blk.20.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 193/ 363]                 blk.21.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 194/ 363]                 blk.21.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 195/ 363]                 blk.21.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 363]            blk.21.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 363]               blk.21.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 363]               blk.21.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 363]                 blk.21.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 363]              blk.21.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 201/ 363]               blk.21.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 202/ 363]                 blk.22.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 203/ 363]                 blk.22.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 204/ 363]                 blk.22.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 205/ 363]            blk.22.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 363]               blk.22.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 363]               blk.22.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 363]                 blk.22.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 363]              blk.22.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 210/ 363]               blk.22.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 211/ 363]                 blk.23.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 212/ 363]                 blk.23.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 213/ 363]                 blk.23.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 214/ 363]            blk.23.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 215/ 363]               blk.23.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 363]               blk.23.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 363]                 blk.23.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 363]              blk.23.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 219/ 363]               blk.23.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 220/ 363]                 blk.24.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 221/ 363]                 blk.24.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 222/ 363]                 blk.24.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 363]            blk.24.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 224/ 363]               blk.24.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 363]               blk.24.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 363]                 blk.24.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 363]              blk.24.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 228/ 363]               blk.24.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 229/ 363]                 blk.25.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 363]                 blk.25.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 231/ 363]                 blk.25.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 363]            blk.25.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 233/ 363]               blk.25.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 363]               blk.25.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 363]                 blk.25.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 363]              blk.25.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 237/ 363]               blk.25.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 238/ 363]                 blk.26.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 239/ 363]                 blk.26.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 240/ 363]                 blk.26.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 241/ 363]            blk.26.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 242/ 363]               blk.26.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 363]               blk.26.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 363]                 blk.26.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 363]              blk.26.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 246/ 363]               blk.26.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 247/ 363]                 blk.27.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 248/ 363]                 blk.27.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 249/ 363]                 blk.27.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 363]            blk.27.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 363]               blk.27.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 363]               blk.27.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 363]                 blk.27.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 363]              blk.27.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 255/ 363]               blk.27.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 256/ 363]                 blk.28.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 257/ 363]                 blk.28.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 258/ 363]                 blk.28.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 363]            blk.28.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 260/ 363]               blk.28.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 363]               blk.28.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 262/ 363]                 blk.28.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 363]              blk.28.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 264/ 363]               blk.28.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 265/ 363]                 blk.29.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 363]                 blk.29.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 267/ 363]                 blk.29.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 363]            blk.29.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 269/ 363]               blk.29.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 363]               blk.29.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 271/ 363]                 blk.29.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 363]              blk.29.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 273/ 363]               blk.29.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 274/ 363]                 blk.30.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 275/ 363]                 blk.30.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 276/ 363]                 blk.30.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 277/ 363]            blk.30.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 363]               blk.30.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 363]               blk.30.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 280/ 363]                 blk.30.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 363]              blk.30.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 282/ 363]               blk.30.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 283/ 363]                 blk.31.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 284/ 363]                 blk.31.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 285/ 363]                 blk.31.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 286/ 363]            blk.31.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 287/ 363]               blk.31.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 363]               blk.31.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 289/ 363]                 blk.31.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 290/ 363]              blk.31.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 291/ 363]               blk.31.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 292/ 363]                 blk.32.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 293/ 363]                 blk.32.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 294/ 363]                 blk.32.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 295/ 363]            blk.32.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 296/ 363]               blk.32.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 297/ 363]               blk.32.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 298/ 363]                 blk.32.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 299/ 363]              blk.32.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 300/ 363]               blk.32.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 301/ 363]                 blk.33.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 302/ 363]                 blk.33.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 303/ 363]                 blk.33.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 304/ 363]            blk.33.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 305/ 363]               blk.33.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 306/ 363]               blk.33.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 307/ 363]                 blk.33.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 308/ 363]              blk.33.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 309/ 363]               blk.33.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 310/ 363]                 blk.34.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 311/ 363]                 blk.34.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 312/ 363]                 blk.34.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 313/ 363]            blk.34.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 314/ 363]               blk.34.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 315/ 363]               blk.34.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 316/ 363]                 blk.34.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 317/ 363]              blk.34.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 318/ 363]               blk.34.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 319/ 363]                 blk.35.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 320/ 363]                 blk.35.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 321/ 363]                 blk.35.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 322/ 363]            blk.35.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 323/ 363]               blk.35.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 324/ 363]               blk.35.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 325/ 363]                 blk.35.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 326/ 363]              blk.35.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 327/ 363]               blk.35.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 328/ 363]                 blk.36.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 329/ 363]                 blk.36.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 330/ 363]                 blk.36.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 331/ 363]            blk.36.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 332/ 363]               blk.36.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 333/ 363]               blk.36.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 334/ 363]                 blk.36.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 335/ 363]              blk.36.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 336/ 363]               blk.36.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 337/ 363]                 blk.37.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 338/ 363]                 blk.37.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 339/ 363]                 blk.37.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 340/ 363]            blk.37.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 341/ 363]               blk.37.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 342/ 363]               blk.37.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 343/ 363]                 blk.37.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 344/ 363]              blk.37.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 345/ 363]               blk.37.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 346/ 363]                 blk.38.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 347/ 363]                 blk.38.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 348/ 363]                 blk.38.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 349/ 363]            blk.38.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 350/ 363]               blk.38.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 351/ 363]               blk.38.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 352/ 363]                 blk.38.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 353/ 363]              blk.38.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 354/ 363]               blk.38.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 355/ 363]                 blk.39.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 356/ 363]                 blk.39.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 357/ 363]                 blk.39.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.120 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 358/ 363]            blk.39.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.096 0.113 0.121 0.113 0.096 0.076 0.055 0.038 0.025 0.021 \n",
      "[ 359/ 363]               blk.39.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 360/ 363]               blk.39.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.076 0.098 0.115 0.122 0.115 0.098 0.076 0.054 0.037 0.024 0.020 \n",
      "[ 361/ 363]                 blk.39.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 362/ 363]              blk.39.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 363/ 363]               blk.39.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "llama_model_quantize_internal: model size  = 24826.58 MB\n",
      "llama_model_quantize_internal: quant size  =  7023.90 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 28706.33 ms\n",
      "main:    total time = 28706.33 ms\n",
      "main: build = 1671 (8fe03ff)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/70B-v2/ggml-model-f16.gguf' to './models/70B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type  f16:  562 tensors\n",
      "llama_model_quantize_internal: meta size = 1740160 bytes\n",
      "[   1/ 723]                    token_embd.weight - [ 8192, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   500.00 MiB ->   140.62 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[   2/ 723]                   output_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[   3/ 723]                        output.weight - [ 8192, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   500.00 MiB ->   205.08 MiB | hist: \n",
      "[   4/ 723]                  blk.0.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.034 0.009 0.014 0.023 0.037 0.059 0.093 0.147 0.198 0.148 0.093 0.059 0.037 0.023 0.014 0.012 \n",
      "[   5/ 723]                  blk.0.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.034 0.008 0.013 0.021 0.035 0.057 0.094 0.153 0.201 0.153 0.094 0.057 0.035 0.021 0.013 0.011 \n",
      "[   6/ 723]                  blk.0.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.096 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[   7/ 723]             blk.0.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.022 0.034 0.052 0.074 0.099 0.120 0.128 0.120 0.099 0.075 0.052 0.035 0.022 0.018 \n",
      "[   8/ 723]                blk.0.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   9/ 723]                blk.0.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.075 0.098 0.117 0.125 0.117 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  10/ 723]                  blk.0.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  11/ 723]               blk.0.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  12/ 723]                blk.0.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  13/ 723]                  blk.1.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.035 0.011 0.017 0.028 0.043 0.066 0.099 0.137 0.160 0.137 0.099 0.066 0.043 0.028 0.017 0.015 \n",
      "[  14/ 723]                  blk.1.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.073 0.099 0.124 0.135 0.124 0.099 0.073 0.050 0.033 0.021 0.018 \n",
      "[  15/ 723]                  blk.1.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.022 0.033 0.050 0.071 0.097 0.124 0.137 0.124 0.097 0.071 0.050 0.033 0.022 0.018 \n",
      "[  16/ 723]             blk.1.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.117 0.098 0.076 0.054 0.036 0.023 0.019 \n",
      "[  17/ 723]                blk.1.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 723]                blk.1.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 723]                  blk.1.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 723]               blk.1.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  21/ 723]                blk.1.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  22/ 723]                  blk.2.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.022 0.035 0.052 0.075 0.099 0.119 0.127 0.119 0.099 0.075 0.053 0.035 0.022 0.018 \n",
      "[  23/ 723]                  blk.2.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.117 0.125 0.117 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  24/ 723]                  blk.2.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  25/ 723]             blk.2.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  26/ 723]                blk.2.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 723]                blk.2.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 723]                  blk.2.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 723]               blk.2.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  30/ 723]                blk.2.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  31/ 723]                  blk.3.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  32/ 723]                  blk.3.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  33/ 723]                  blk.3.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  34/ 723]             blk.3.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  35/ 723]                blk.3.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 723]                blk.3.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  37/ 723]                  blk.3.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 723]               blk.3.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  39/ 723]                blk.3.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  40/ 723]                  blk.4.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[  41/ 723]                  blk.4.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.024 0.020 \n",
      "[  42/ 723]                  blk.4.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.024 0.020 \n",
      "[  43/ 723]             blk.4.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  44/ 723]                blk.4.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 723]                blk.4.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  46/ 723]                  blk.4.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  47/ 723]               blk.4.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  48/ 723]                blk.4.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  49/ 723]                  blk.5.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  50/ 723]                  blk.5.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 723]                  blk.5.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  52/ 723]             blk.5.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  53/ 723]                blk.5.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 723]                blk.5.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  55/ 723]                  blk.5.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  56/ 723]               blk.5.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  57/ 723]                blk.5.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  58/ 723]                  blk.6.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  59/ 723]                  blk.6.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  60/ 723]                  blk.6.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[  61/ 723]             blk.6.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  62/ 723]                blk.6.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 723]                blk.6.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 723]                  blk.6.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 723]               blk.6.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  66/ 723]                blk.6.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  67/ 723]                  blk.7.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  68/ 723]                  blk.7.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  69/ 723]                  blk.7.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  70/ 723]             blk.7.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  71/ 723]                blk.7.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 723]                blk.7.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  73/ 723]                  blk.7.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 723]               blk.7.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  75/ 723]                blk.7.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  76/ 723]                  blk.8.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 723]                  blk.8.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 723]                  blk.8.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  79/ 723]             blk.8.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 723]                blk.8.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 723]                blk.8.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  82/ 723]                  blk.8.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  83/ 723]               blk.8.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  84/ 723]                blk.8.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  85/ 723]                  blk.9.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 723]                  blk.9.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  87/ 723]                  blk.9.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  88/ 723]             blk.9.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  89/ 723]                blk.9.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 723]                blk.9.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  91/ 723]                  blk.9.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  92/ 723]               blk.9.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  93/ 723]                blk.9.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  94/ 723]                 blk.10.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  95/ 723]                 blk.10.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  96/ 723]                 blk.10.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  97/ 723]            blk.10.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  98/ 723]               blk.10.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 723]               blk.10.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 100/ 723]                 blk.10.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 101/ 723]              blk.10.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 102/ 723]               blk.10.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 103/ 723]                 blk.11.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 723]                 blk.11.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 723]                 blk.11.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 106/ 723]            blk.11.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 107/ 723]               blk.11.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 723]               blk.11.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 109/ 723]                 blk.11.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 110/ 723]              blk.11.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 111/ 723]               blk.11.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 112/ 723]                 blk.12.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 113/ 723]                 blk.12.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 723]                 blk.12.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 723]            blk.12.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 116/ 723]               blk.12.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 723]               blk.12.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 118/ 723]                 blk.12.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 723]              blk.12.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 120/ 723]               blk.12.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 121/ 723]                 blk.13.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 723]                 blk.13.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 123/ 723]                 blk.13.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 723]            blk.13.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 125/ 723]               blk.13.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 723]               blk.13.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 127/ 723]                 blk.13.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 723]              blk.13.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 129/ 723]               blk.13.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 130/ 723]                 blk.14.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 131/ 723]                 blk.14.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 723]                 blk.14.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 723]            blk.14.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 134/ 723]               blk.14.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 723]               blk.14.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 136/ 723]                 blk.14.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 723]              blk.14.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 138/ 723]               blk.14.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 139/ 723]                 blk.15.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 723]                 blk.15.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 141/ 723]                 blk.15.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 723]            blk.15.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 143/ 723]               blk.15.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 723]               blk.15.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 145/ 723]                 blk.15.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 723]              blk.15.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 147/ 723]               blk.15.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 148/ 723]                 blk.16.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 723]                 blk.16.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 150/ 723]                 blk.16.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 723]            blk.16.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 152/ 723]               blk.16.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 723]               blk.16.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 154/ 723]                 blk.16.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 723]              blk.16.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 156/ 723]               blk.16.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 157/ 723]                 blk.17.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 723]                 blk.17.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 159/ 723]                 blk.17.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 723]            blk.17.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 161/ 723]               blk.17.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 723]               blk.17.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 163/ 723]                 blk.17.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 723]              blk.17.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 165/ 723]               blk.17.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 166/ 723]                 blk.18.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 167/ 723]                 blk.18.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 168/ 723]                 blk.18.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 723]            blk.18.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 170/ 723]               blk.18.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 723]               blk.18.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 172/ 723]                 blk.18.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 723]              blk.18.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 174/ 723]               blk.18.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 175/ 723]                 blk.19.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 176/ 723]                 blk.19.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 177/ 723]                 blk.19.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 723]            blk.19.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 179/ 723]               blk.19.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 723]               blk.19.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 723]                 blk.19.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 723]              blk.19.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 183/ 723]               blk.19.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 184/ 723]                 blk.20.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 723]                 blk.20.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 186/ 723]                 blk.20.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 723]            blk.20.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 188/ 723]               blk.20.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 723]               blk.20.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 723]                 blk.20.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 723]              blk.20.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 192/ 723]               blk.20.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 193/ 723]                 blk.21.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 194/ 723]                 blk.21.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 195/ 723]                 blk.21.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 723]            blk.21.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 723]               blk.21.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 723]               blk.21.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 723]                 blk.21.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 723]              blk.21.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 201/ 723]               blk.21.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 202/ 723]                 blk.22.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 203/ 723]                 blk.22.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 204/ 723]                 blk.22.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 205/ 723]            blk.22.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 723]               blk.22.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 723]               blk.22.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 723]                 blk.22.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 723]              blk.22.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 210/ 723]               blk.22.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 211/ 723]                 blk.23.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 212/ 723]                 blk.23.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 213/ 723]                 blk.23.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 214/ 723]            blk.23.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 215/ 723]               blk.23.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 723]               blk.23.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 723]                 blk.23.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 723]              blk.23.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 219/ 723]               blk.23.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 220/ 723]                 blk.24.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 221/ 723]                 blk.24.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 222/ 723]                 blk.24.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 723]            blk.24.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 224/ 723]               blk.24.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 723]               blk.24.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 723]                 blk.24.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 723]              blk.24.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 228/ 723]               blk.24.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 229/ 723]                 blk.25.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 723]                 blk.25.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 231/ 723]                 blk.25.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 723]            blk.25.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 233/ 723]               blk.25.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 723]               blk.25.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 723]                 blk.25.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 723]              blk.25.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 237/ 723]               blk.25.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 238/ 723]                 blk.26.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 239/ 723]                 blk.26.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 240/ 723]                 blk.26.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 241/ 723]            blk.26.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 242/ 723]               blk.26.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 723]               blk.26.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 723]                 blk.26.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 723]              blk.26.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 246/ 723]               blk.26.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 247/ 723]                 blk.27.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 248/ 723]                 blk.27.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 249/ 723]                 blk.27.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 723]            blk.27.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 723]               blk.27.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 723]               blk.27.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 723]                 blk.27.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 723]              blk.27.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 255/ 723]               blk.27.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 256/ 723]                 blk.28.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 257/ 723]                 blk.28.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 258/ 723]                 blk.28.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 723]            blk.28.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 260/ 723]               blk.28.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 723]               blk.28.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 262/ 723]                 blk.28.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 723]              blk.28.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 264/ 723]               blk.28.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 265/ 723]                 blk.29.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 723]                 blk.29.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 267/ 723]                 blk.29.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 723]            blk.29.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 269/ 723]               blk.29.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 723]               blk.29.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 271/ 723]                 blk.29.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 723]              blk.29.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 273/ 723]               blk.29.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 274/ 723]                 blk.30.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 275/ 723]                 blk.30.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 276/ 723]                 blk.30.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 277/ 723]            blk.30.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 723]               blk.30.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 723]               blk.30.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 280/ 723]                 blk.30.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 723]              blk.30.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 282/ 723]               blk.30.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 283/ 723]                 blk.31.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 284/ 723]                 blk.31.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 285/ 723]                 blk.31.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 286/ 723]            blk.31.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 287/ 723]               blk.31.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 723]               blk.31.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 289/ 723]                 blk.31.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 290/ 723]              blk.31.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 291/ 723]               blk.31.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 292/ 723]                 blk.32.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 293/ 723]                 blk.32.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 294/ 723]                 blk.32.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 295/ 723]            blk.32.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 296/ 723]               blk.32.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 297/ 723]               blk.32.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 298/ 723]                 blk.32.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 299/ 723]              blk.32.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 300/ 723]               blk.32.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 301/ 723]                 blk.33.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 302/ 723]                 blk.33.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 303/ 723]                 blk.33.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 304/ 723]            blk.33.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 305/ 723]               blk.33.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 306/ 723]               blk.33.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 307/ 723]                 blk.33.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 308/ 723]              blk.33.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 309/ 723]               blk.33.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 310/ 723]                 blk.34.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 311/ 723]                 blk.34.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 312/ 723]                 blk.34.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 313/ 723]            blk.34.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 314/ 723]               blk.34.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 315/ 723]               blk.34.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 316/ 723]                 blk.34.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 317/ 723]              blk.34.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 318/ 723]               blk.34.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 319/ 723]                 blk.35.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 320/ 723]                 blk.35.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 321/ 723]                 blk.35.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 322/ 723]            blk.35.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 323/ 723]               blk.35.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 324/ 723]               blk.35.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 325/ 723]                 blk.35.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 326/ 723]              blk.35.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 327/ 723]               blk.35.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 328/ 723]                 blk.36.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 329/ 723]                 blk.36.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 330/ 723]                 blk.36.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 331/ 723]            blk.36.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 332/ 723]               blk.36.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 333/ 723]               blk.36.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 334/ 723]                 blk.36.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 335/ 723]              blk.36.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 336/ 723]               blk.36.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 337/ 723]                 blk.37.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 338/ 723]                 blk.37.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 339/ 723]                 blk.37.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.076 0.057 0.039 0.025 0.021 \n",
      "[ 340/ 723]            blk.37.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 341/ 723]               blk.37.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 342/ 723]               blk.37.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 343/ 723]                 blk.37.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 344/ 723]              blk.37.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 345/ 723]               blk.37.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 346/ 723]                 blk.38.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 347/ 723]                 blk.38.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 348/ 723]                 blk.38.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 349/ 723]            blk.38.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 350/ 723]               blk.38.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 351/ 723]               blk.38.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 352/ 723]                 blk.38.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 353/ 723]              blk.38.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 354/ 723]               blk.38.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 355/ 723]                 blk.39.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 356/ 723]                 blk.39.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 357/ 723]                 blk.39.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 358/ 723]            blk.39.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 359/ 723]               blk.39.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 360/ 723]               blk.39.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 361/ 723]                 blk.39.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 362/ 723]              blk.39.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 363/ 723]               blk.39.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 364/ 723]                 blk.40.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 365/ 723]                 blk.40.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 366/ 723]                 blk.40.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 367/ 723]            blk.40.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 368/ 723]               blk.40.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 369/ 723]               blk.40.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 370/ 723]                 blk.40.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 371/ 723]              blk.40.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 372/ 723]               blk.40.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 373/ 723]                 blk.41.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 374/ 723]                 blk.41.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 375/ 723]                 blk.41.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 376/ 723]            blk.41.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 377/ 723]               blk.41.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 378/ 723]               blk.41.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 379/ 723]                 blk.41.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 380/ 723]              blk.41.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 381/ 723]               blk.41.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 382/ 723]                 blk.42.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 383/ 723]                 blk.42.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 384/ 723]                 blk.42.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 385/ 723]            blk.42.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 386/ 723]               blk.42.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 387/ 723]               blk.42.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 388/ 723]                 blk.42.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 389/ 723]              blk.42.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 390/ 723]               blk.42.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 391/ 723]                 blk.43.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 392/ 723]                 blk.43.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 393/ 723]                 blk.43.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 394/ 723]            blk.43.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 395/ 723]               blk.43.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 396/ 723]               blk.43.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 397/ 723]                 blk.43.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 398/ 723]              blk.43.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 399/ 723]               blk.43.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 400/ 723]                 blk.44.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 401/ 723]                 blk.44.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 402/ 723]                 blk.44.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 403/ 723]            blk.44.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 404/ 723]               blk.44.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 405/ 723]               blk.44.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 406/ 723]                 blk.44.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 407/ 723]              blk.44.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 408/ 723]               blk.44.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 409/ 723]                 blk.45.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 410/ 723]                 blk.45.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 411/ 723]                 blk.45.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 412/ 723]            blk.45.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 413/ 723]               blk.45.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 414/ 723]               blk.45.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 415/ 723]                 blk.45.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 416/ 723]              blk.45.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 417/ 723]               blk.45.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 418/ 723]                 blk.46.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 419/ 723]                 blk.46.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 420/ 723]                 blk.46.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 421/ 723]            blk.46.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 422/ 723]               blk.46.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 423/ 723]               blk.46.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 424/ 723]                 blk.46.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 425/ 723]              blk.46.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 426/ 723]               blk.46.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 427/ 723]                 blk.47.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 428/ 723]                 blk.47.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.039 0.025 0.020 \n",
      "[ 429/ 723]                 blk.47.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 430/ 723]            blk.47.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 431/ 723]               blk.47.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 432/ 723]               blk.47.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 433/ 723]                 blk.47.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 434/ 723]              blk.47.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 435/ 723]               blk.47.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 436/ 723]                 blk.48.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 437/ 723]                 blk.48.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[ 438/ 723]                 blk.48.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 439/ 723]            blk.48.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 440/ 723]               blk.48.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 441/ 723]               blk.48.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 442/ 723]                 blk.48.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 443/ 723]              blk.48.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 444/ 723]               blk.48.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 445/ 723]                 blk.49.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 446/ 723]                 blk.49.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 447/ 723]                 blk.49.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 448/ 723]            blk.49.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 449/ 723]               blk.49.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 450/ 723]               blk.49.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 451/ 723]                 blk.49.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 452/ 723]              blk.49.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 453/ 723]               blk.49.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 454/ 723]                 blk.50.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 455/ 723]                 blk.50.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.123 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 456/ 723]                 blk.50.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 457/ 723]            blk.50.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 458/ 723]               blk.50.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 459/ 723]               blk.50.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 460/ 723]                 blk.50.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 461/ 723]              blk.50.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 462/ 723]               blk.50.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 463/ 723]                 blk.51.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 464/ 723]                 blk.51.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 465/ 723]                 blk.51.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 466/ 723]            blk.51.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 467/ 723]               blk.51.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 468/ 723]               blk.51.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 469/ 723]                 blk.51.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 470/ 723]              blk.51.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 471/ 723]               blk.51.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 472/ 723]                 blk.52.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 473/ 723]                 blk.52.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 474/ 723]                 blk.52.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 475/ 723]            blk.52.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 476/ 723]               blk.52.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 477/ 723]               blk.52.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 478/ 723]                 blk.52.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 479/ 723]              blk.52.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 480/ 723]               blk.52.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 481/ 723]                 blk.53.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 482/ 723]                 blk.53.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 483/ 723]                 blk.53.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 484/ 723]            blk.53.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 485/ 723]               blk.53.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 486/ 723]               blk.53.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 487/ 723]                 blk.53.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 488/ 723]              blk.53.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 489/ 723]               blk.53.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 490/ 723]                 blk.54.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 491/ 723]                 blk.54.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 492/ 723]                 blk.54.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 493/ 723]            blk.54.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 494/ 723]               blk.54.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 495/ 723]               blk.54.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 496/ 723]                 blk.54.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 497/ 723]              blk.54.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 498/ 723]               blk.54.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 499/ 723]                 blk.55.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 500/ 723]                 blk.55.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 501/ 723]                 blk.55.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 502/ 723]            blk.55.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 503/ 723]               blk.55.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 504/ 723]               blk.55.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 505/ 723]                 blk.55.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 506/ 723]              blk.55.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 507/ 723]               blk.55.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 508/ 723]                 blk.56.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 509/ 723]                 blk.56.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 510/ 723]                 blk.56.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 511/ 723]            blk.56.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 512/ 723]               blk.56.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 513/ 723]               blk.56.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 514/ 723]                 blk.56.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 515/ 723]              blk.56.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 516/ 723]               blk.56.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 517/ 723]                 blk.57.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 518/ 723]                 blk.57.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 519/ 723]                 blk.57.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 520/ 723]            blk.57.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 521/ 723]               blk.57.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 522/ 723]               blk.57.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 523/ 723]                 blk.57.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 524/ 723]              blk.57.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 525/ 723]               blk.57.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 526/ 723]                 blk.58.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 527/ 723]                 blk.58.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.097 0.117 0.126 0.117 0.097 0.075 0.054 0.037 0.023 0.019 \n",
      "[ 528/ 723]                 blk.58.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 529/ 723]            blk.58.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 530/ 723]               blk.58.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 531/ 723]               blk.58.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 532/ 723]                 blk.58.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 533/ 723]              blk.58.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 534/ 723]               blk.58.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 535/ 723]                 blk.59.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 536/ 723]                 blk.59.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.037 0.054 0.075 0.097 0.116 0.125 0.116 0.097 0.075 0.054 0.037 0.024 0.019 \n",
      "[ 537/ 723]                 blk.59.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 538/ 723]            blk.59.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 539/ 723]               blk.59.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 540/ 723]               blk.59.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 541/ 723]                 blk.59.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 542/ 723]              blk.59.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 543/ 723]               blk.59.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 544/ 723]                 blk.60.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 545/ 723]                 blk.60.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.072 0.097 0.123 0.140 0.123 0.097 0.072 0.050 0.034 0.021 0.018 \n",
      "[ 546/ 723]                 blk.60.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 547/ 723]            blk.60.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 548/ 723]               blk.60.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 549/ 723]               blk.60.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 550/ 723]                 blk.60.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 551/ 723]              blk.60.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 552/ 723]               blk.60.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 553/ 723]                 blk.61.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 554/ 723]                 blk.61.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.118 0.129 0.118 0.097 0.074 0.053 0.036 0.023 0.019 \n",
      "[ 555/ 723]                 blk.61.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 556/ 723]            blk.61.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 557/ 723]               blk.61.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 558/ 723]               blk.61.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 559/ 723]                 blk.61.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 560/ 723]              blk.61.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 561/ 723]               blk.61.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 562/ 723]                 blk.62.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 563/ 723]                 blk.62.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.118 0.129 0.118 0.097 0.074 0.053 0.036 0.023 0.019 \n",
      "[ 564/ 723]                 blk.62.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 565/ 723]            blk.62.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 566/ 723]               blk.62.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 567/ 723]               blk.62.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 568/ 723]                 blk.62.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 569/ 723]              blk.62.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 570/ 723]               blk.62.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 571/ 723]                 blk.63.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 572/ 723]                 blk.63.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.035 0.052 0.074 0.097 0.119 0.132 0.119 0.097 0.074 0.053 0.035 0.023 0.019 \n",
      "[ 573/ 723]                 blk.63.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 574/ 723]            blk.63.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 575/ 723]               blk.63.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 576/ 723]               blk.63.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 577/ 723]                 blk.63.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 578/ 723]              blk.63.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 579/ 723]               blk.63.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 580/ 723]                 blk.64.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 581/ 723]                 blk.64.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.036 0.054 0.075 0.097 0.117 0.125 0.117 0.097 0.075 0.054 0.037 0.024 0.019 \n",
      "[ 582/ 723]                 blk.64.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 583/ 723]            blk.64.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 584/ 723]               blk.64.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 585/ 723]               blk.64.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 586/ 723]                 blk.64.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 587/ 723]              blk.64.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 588/ 723]               blk.64.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 589/ 723]                 blk.65.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.115 0.122 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 590/ 723]                 blk.65.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.022 0.034 0.051 0.072 0.097 0.122 0.138 0.122 0.097 0.072 0.051 0.034 0.022 0.018 \n",
      "[ 591/ 723]                 blk.65.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 592/ 723]            blk.65.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 593/ 723]               blk.65.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 594/ 723]               blk.65.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 595/ 723]                 blk.65.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 596/ 723]              blk.65.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 597/ 723]               blk.65.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 598/ 723]                 blk.66.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 599/ 723]                 blk.66.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.119 0.130 0.119 0.097 0.074 0.053 0.035 0.023 0.019 \n",
      "[ 600/ 723]                 blk.66.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 601/ 723]            blk.66.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 602/ 723]               blk.66.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 603/ 723]               blk.66.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 604/ 723]                 blk.66.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 605/ 723]              blk.66.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 606/ 723]               blk.66.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 607/ 723]                 blk.67.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.097 0.116 0.125 0.117 0.097 0.075 0.054 0.036 0.023 0.019 \n",
      "[ 608/ 723]                 blk.67.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.035 0.053 0.074 0.097 0.119 0.133 0.119 0.096 0.073 0.052 0.035 0.023 0.019 \n",
      "[ 609/ 723]                 blk.67.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 610/ 723]            blk.67.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 611/ 723]               blk.67.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 612/ 723]               blk.67.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 613/ 723]                 blk.67.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 614/ 723]              blk.67.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 615/ 723]               blk.67.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 616/ 723]                 blk.68.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 617/ 723]                 blk.68.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 618/ 723]                 blk.68.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 619/ 723]            blk.68.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 620/ 723]               blk.68.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 621/ 723]               blk.68.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 622/ 723]                 blk.68.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 623/ 723]              blk.68.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 624/ 723]               blk.68.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 625/ 723]                 blk.69.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 626/ 723]                 blk.69.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 627/ 723]                 blk.69.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 628/ 723]            blk.69.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 629/ 723]               blk.69.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 630/ 723]               blk.69.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 631/ 723]                 blk.69.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 632/ 723]              blk.69.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 633/ 723]               blk.69.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 634/ 723]                 blk.70.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 635/ 723]                 blk.70.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 636/ 723]                 blk.70.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 637/ 723]            blk.70.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 638/ 723]               blk.70.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 639/ 723]               blk.70.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 640/ 723]                 blk.70.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 641/ 723]              blk.70.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 642/ 723]               blk.70.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 643/ 723]                 blk.71.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 644/ 723]                 blk.71.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 645/ 723]                 blk.71.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 646/ 723]            blk.71.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 647/ 723]               blk.71.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 648/ 723]               blk.71.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 649/ 723]                 blk.71.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 650/ 723]              blk.71.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 651/ 723]               blk.71.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 652/ 723]                 blk.72.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 653/ 723]                 blk.72.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 654/ 723]                 blk.72.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 655/ 723]            blk.72.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 656/ 723]               blk.72.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 657/ 723]               blk.72.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 658/ 723]                 blk.72.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 659/ 723]              blk.72.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 660/ 723]               blk.72.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 661/ 723]                 blk.73.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 662/ 723]                 blk.73.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 663/ 723]                 blk.73.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 664/ 723]            blk.73.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 665/ 723]               blk.73.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 666/ 723]               blk.73.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 667/ 723]                 blk.73.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 668/ 723]              blk.73.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 669/ 723]               blk.73.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 670/ 723]                 blk.74.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 671/ 723]                 blk.74.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 672/ 723]                 blk.74.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 673/ 723]            blk.74.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 674/ 723]               blk.74.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 675/ 723]               blk.74.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 676/ 723]                 blk.74.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 677/ 723]              blk.74.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 678/ 723]               blk.74.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 679/ 723]                 blk.75.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 680/ 723]                 blk.75.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 681/ 723]                 blk.75.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 682/ 723]            blk.75.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 683/ 723]               blk.75.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 684/ 723]               blk.75.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 685/ 723]                 blk.75.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 686/ 723]              blk.75.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 687/ 723]               blk.75.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 688/ 723]                 blk.76.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 689/ 723]                 blk.76.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 690/ 723]                 blk.76.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 691/ 723]            blk.76.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 692/ 723]               blk.76.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 693/ 723]               blk.76.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 694/ 723]                 blk.76.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 695/ 723]              blk.76.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 696/ 723]               blk.76.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 697/ 723]                 blk.77.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 698/ 723]                 blk.77.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 699/ 723]                 blk.77.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 700/ 723]            blk.77.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.040 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.040 0.026 0.021 \n",
      "[ 701/ 723]               blk.77.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 702/ 723]               blk.77.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 703/ 723]                 blk.77.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 704/ 723]              blk.77.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 705/ 723]               blk.77.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 706/ 723]                 blk.78.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 707/ 723]                 blk.78.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 708/ 723]                 blk.78.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 709/ 723]            blk.78.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 710/ 723]               blk.78.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 711/ 723]               blk.78.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 712/ 723]                 blk.78.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 713/ 723]              blk.78.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 714/ 723]               blk.78.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 715/ 723]                 blk.79.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 716/ 723]                 blk.79.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 717/ 723]                 blk.79.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 718/ 723]            blk.79.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.040 0.026 0.021 \n",
      "[ 719/ 723]               blk.79.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 720/ 723]               blk.79.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 721/ 723]                 blk.79.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 722/ 723]              blk.79.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 723/ 723]               blk.79.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "llama_model_quantize_internal: model size  = 131565.03 MB\n",
      "llama_model_quantize_internal: quant size  = 37070.73 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 105771.41 ms\n",
      "main:    total time = 105771.41 ms\n"
     ]
    }
   ],
   "source": [
    "# quantize the model to 4-bits (using q4_0 method)\n",
    "!./quantize ./models/7B-v2/ggml-model-f16.gguf ./models/7B-v2/ggml-model-q4_0.gguf q4_0\n",
    "!./quantize ./models/13B-v2/ggml-model-f16.gguf ./models/13B-v2/ggml-model-q4_0.gguf q4_0\n",
    "!./quantize ./models/70B-v2/ggml-model-f16.gguf ./models/70B-v2/ggml-model-q4_0.gguf q4_0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "416ca561-de1a-4094-ae0b-fd71408d45e6",
   "metadata": {},
   "source": [
    "# inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0aede9-2f19-41e1-bb4f-1a1d30a00156",
   "metadata": {},
   "source": [
    "### 7B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4c50d2ab-fc82-4119-8ac3-38ead2b8fee8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703325941\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her star, she conquered and peopled half of Europe; for the gash in her flesh from the Thirty Years War time, had been cured and shut up by the French Revolution. It was as if Nature, having intended France to be the fashion model of the world, had too much exalted her. She ran like a strong man in a storm, with unbound hair and uplifted arms. \n",
      "It was universally admitted, I mean, that if you wanted to find out what any particular civilized person in any walk of life really believed, or, failing this, pretended to believe, you had to examine his daily press. And even then there were precautions. On any subject you especially desired to know about, it was far more dangerous to consult the newspaper on your own side than on the other: much more dangerous, and (strictly in accordance with what is called The Prussian Method) not a little more instructive. You could only find out the truth of big matters by exhaustive enquiry into every detail of life, and by keeping all your apparatus on the qui vive for any speck or blemish in those details which might be signs of what was passing unnoticed elsewhere. And, of course, the speck or the blemish once found must itself be thoroughly examined, until at last (by a very long circuit and by means of the most delicate-sounding ratiocination) it came to be represented as part of some one definite theory. Meanwhile you went on reading your newspaper. But there were things which could not be known in this way: for instance, whether or not the police had found out that a certain person had murdered someone else; and whether or not any particular young lady was going to accept her lover's ultimatum about marrying him on a fixed date, with or without making the necessary arrangements about selling her clothes and wearing nothing but linen. It must be said fair-mindedly that it was in the main through no fault of their own that some people went wrong. The chief defect lay not so much in what these people did, as in what they thought. This is an important distinction: for it is by their thoughts, and not by their actions or inactions, that many bad people lead such very miserable lives. One can have nothing but sympathy for poor Mrs. Moore. Her heart was broken by the misfortunes of her son, Basil. And this, remember, happened even before he became an embezzler and a thief: and if that wasn't enough to break one's heart, there was also the business with his daughter's marriage and all that followed afterwards. Of course it must be admitted at once that she took all these things much more to heart than need have been necessary--she gave herself an ulcer on her right thigh for instance: but this is only because she allowed herself to think of them all the time, whereas one would rather hope that most people have very little trouble thinking about them. She could have made things much easier for herself if she'd simply said to herself every day of her life \"It doesn't matter a damn,\" and then proceeded at once to devote herself unremittingly to some objectively important task or other--such, for instance, as writing a book on the subject of \"How to get an Ulcer in your thigh.\" But Mrs. Moore couldn't manage that, any more than I can, because she wasn't strong-willed enough, and I, alas! am too weak-willed and idle even to try: so we both remain miserable for the rest of our lives.\n",
      "There are innumerable little difficulties, difficulties with which most people cope without feeling particularly unhappy about them--such as being bored or not getting your own way--which, if you really look at it, is just a kind of difficulty of that general sort I mean; and that is what Mrs. Moore couldn't manage to get on with in life; so that she was all her life the victim of little difficulties which, for some obscure reason, she simply could not put up with; as one might say \"I can't do this or that: it isn't that I want to do them: it is simply that I can't get round them.\" It makes one rather sad--to think that a person can be in such straits.\n",
      "And there's also, of course, the whole business about her daughter's marriage. Now there's no doubt whatever but that was the greatest difficulty of all: and it had to be so because, if I remember rightly, she herself said afterwards (forgetting of course that she herself caused all this trouble) \"It might have been possible to live with, but you can't bring a young\n",
      "llama_print_timings:        load time =    3212.96 ms\n",
      "llama_print_timings:      sample time =     565.60 ms /  1024 runs   (    0.55 ms per token,  1810.48 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2086.62 ms /   494 tokens (    4.22 ms per token,   236.75 tokens per second)\n",
      "llama_print_timings:        eval time =   22187.28 ms /  1023 runs   (   21.69 ms per token,    46.11 tokens per second)\n",
      "llama_print_timings:       total time =   25148.88 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3e237938-4375-406a-b329-48d6d2363aa9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703325971\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of M. Necker, the finances had been so well pared away, that year by year half the working classes of Paris (and even those) found it impossible to live upon the other half. This was the kind of reforming which disheartened our descendants from America, and caused them to seek for themselves a home in the wilderness beyond the Atlantic. Forbearing from asking, or from expecting, that any State should guarantee to every citizen a weekly bread, the maories of France were obliged to subsist upon the hope of being starved out, and upon the proceeds of a lottery for the benefit of those who had already been starved out. In these circumistances it was calculated by Mr. Gladstone, (who was at that time Prime Minister of England) that eighteen or nineteen millions sterling a-year would be required for the simple maintenance of the French people. After some years of peace and prosperity, they have paid down, within the memory of man, one hundred and seventy-five millions sterling in order to restore their former happiness! \n",
      " Mr. Crawford, who has had occasion to observe such matters very closely indeed, tells us that England has no monied aristocracy. This we are willing to take upon trust: for we ourselves have never had any experience in the matter, and can give our own testimony to the contrary. When Mr. Crawford says that this is so because there is no aristocratic class in England (except the nobility) he must mean to imply that there is no class in the community who are disproportionately rich: whereas in truth the English monied aristocracy, when they come into contact with other classes of men, always assume an air which puts them, so far as we can make out by observation, in a very favourable light. That this is really no more than what the circumstances admit is shown by what Mr. Crawford goes on to say: namely that there are among the English aristocracy a number of families whose wealth, inherited from many generations past, amounts to immense sums of money. It will be observed in passing that if this assertion is correct we should not expect to find very much left of that family pride which was so much talked of by Mr. Crawford's friend. As it is, we see nothing like a good example of any of these aristocratic families, and know for a certainty (for the reason assigned) that there are none. The case of Mr. Crawford himself will illustrate this.\n",
      "The truth seems to be that the English aristocracy in this respect are no better off than the rest of mankind: — it being true as a general rule that men in affluence always have some one to feel themselves at a disadvantage by their wealth and greatness. If a person is rich and powerful, there will almost certainly be some one who feels himself less fortunate than the other in respect to this or that advantage. But when these are all in equal condition and rank (as is the case with the English aristocracy) it may perhaps happen that the man of the highest station among them will not at once feel his own superiority, because he has no one to compare himself with who is less fortunate: whereas if there were only a few members of such families their situation would be very different from what it now is. As it is we see no examples of that class pride which the French are so well-known for among their noblesse — and if we wish to find an example, we must seek one among the middle classes. The reason why this circumstance does not occur more often than at present might possibly be this: That those families whose wealth is inherited have usually no occasion or cause for feeling themselves at a disadvantage from want of pride; whereas the poorer noblesse, having to look after themselves, and being much oftener in circumstances where they must feel themselves at a disadvantage by their poverty, are more apt to be proud: but the fact is that the English aristocracy are all pretty nearly on an equality with one another — so there is no one among them who can say, 'I am not what you think me', or who has any one of his companions upon whom he may rest. And in like manner when a man feels himself superior to his neighbours it seems more from the fact that they are not his equals than from being conscious of their own deficiencies.\n",
      "The only classes among which there is any real pride in the English nation at this moment, seem to be the commercial and professional ones. The commercial class feel themselves superior to the mass of mankind who cannot maintain themselves by industry — and therefore it seems necessary for them to assert that they are more intelligent than other people; but perhaps more of their pride is occasioned by feeling themselves in circumstances where there are a good many things that they\n",
      "llama_print_timings:        load time =    2331.27 ms\n",
      "llama_print_timings:      sample time =     570.65 ms /  1024 runs   (    0.56 ms per token,  1794.45 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2094.79 ms /   494 tokens (    4.24 ms per token,   235.82 tokens per second)\n",
      "llama_print_timings:        eval time =   22211.52 ms /  1023 runs   (   21.71 ms per token,    46.06 tokens per second)\n",
      "llama_print_timings:       total time =   25186.90 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "eb2e9a98-ed63-4b0b-b858-8a69c3cde67f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326000\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her experienced monarch (may he be an everlasting pantomime!), France ensured, by dint of taxation and regulations, that millions of francs should be drained from the country yearly without leaving a scar. Every change of government had prepared the way for the next – there being no end in view but one permanent revolution, in the interests of which every agency, including the sanction and even the action of the supreme governing magistracy, was available. \n",
      " Amongst these economical and financial reforms, monetary sovereignty (the death warrant to the French monarchy) was achieved. In order that the public might be prepared for this economic miracle, it was first made necessary that the money of the nation, whatever the amount might be, must be reduced more than ten per cent, in every year of the Revolutionary Rule. This being done, taxes were so adjusted as almost to deaden entirely all sense of locality in those who paid them, and at the same time were made to fall most heavily on every description of property, whether real or personal. \n",
      " A portion of the public debt was assumed by the Nation; but a great deal more remained, inasmuch as so much money had been borrowed for so many projects with very little hope from results. The annuities were funded, it is true; but the interest on the debt continued to be paid, partly out of the national contributions, and partly out of taxes. In short, the public debt was so regulated as to be always a subject of profound political economy.\n",
      "The Revolution had been accompanied by an immense extinction of property. To this may be added an enormous destruction of productions. It is calculated that 10 millions of francs were sacrificed during the war, and not less than double that sum wasted in the domestic administration of the republic, after the victories were obtained. The result of these prodigies in waste and devastation was to leave the whole country almost wholly depopulated, its trade totally disorganized, and its capital entirely annihilated; a state of things which was not rectified until twenty-five years afterwards, when a reaction set in that did for France what the French Revolution had done for the other nations.\n",
      "CHAPTER II – PHILOSOPHERS OF THE FIRST EPOCH IN 1789\n",
      "The philosophy that had been engendered in Europe in the century previous to the French Revolution was a great and glorious production of the human understanding, but not very well understood or practically applied by those who made it. It was an immense treasure-house of knowledge, of which the philosopher of the time knew only half; it was a splendid theatre full of lights and illuminated with many colours, from which the philosopher sat in solemn silence to behold the play unseen by any one.\n",
      "The philosophy of that period was two-fold: it had been either speculative or practical. In speculation, man had travelled far towards comprehension; he knew more of the constitutions and laws of things than he could have known of their names if he had had all the dictionaries in the world to help him; but when he came to apply this knowledge, he found that it was not adapted for use – there were many obstacles between him and success. He thought that his philosophy would carry him through the difficulties without trouble or fatigue to himself – that, on the contrary, all these difficulties belonged to the world, of which philosophy had no concern; they were things in themselves, but he thought them in connection with his purposes. The philosophers of those days saw their system a mass of brilliant forms, all unsubstantial as clouds, and their life was spent in dreams over it.\n",
      "Those who made this speculative philosophy were men who had gone through the discipline of thinking; they had tried conclusions by themselves – had put the question, 'What is true? What is good?' upon every subject which presented itself to them – upon science as well as upon religion, politics as well as upon morals. They had found out what seemed true in science, and what was false; what seemed to be good in politics, and what evil; what was right in morals, and what wrong; but when they came to make the application of it to life, they did not know where to begin – there was no practical direction or example which could point out the way. The philosophy of that age was a mighty power, but its hand was tied behind it with cords of adamant, and its face was turned away from the world, for it could not see it; it had all the strength of intellect, but not one particle of action in connection with it; like a mighty river which flows unheeded through the w\n",
      "llama_print_timings:        load time =    1714.11 ms\n",
      "llama_print_timings:      sample time =     564.47 ms /  1024 runs   (    0.55 ms per token,  1814.10 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2085.20 ms /   494 tokens (    4.22 ms per token,   236.91 tokens per second)\n",
      "llama_print_timings:        eval time =   22118.35 ms /  1023 runs   (   21.62 ms per token,    46.25 tokens per second)\n",
      "llama_print_timings:       total time =   25077.47 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c40741-4720-463b-a699-a05ae9266ab2",
   "metadata": {},
   "source": [
    "### 7B f16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e3683e44-8c9a-4f85-abfc-37c024375357",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326029\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its glorious Magistrates, the French nation blundered upon new and unheard-of modes of administration; until, at length, arrived at the point of dead relaxation, all the odious economy of an Austrian or Prussian bureaucrat, the whole order of things in every department of public affairs was declared to be involuntarily enslaved. I have ordered that every palace, abbey, and monastery shall be abolished, that no more than three hundred priests may be maintained, that no ecclesiastical processions shall outnumber thirty persons. I do not forget the two millions of francs, distributed among the religious communities: whereof eleven hundred and fifty thousand remain unpaid. In order to satisfy your wishes in this respect, I have caused these eleven hundred and fifty thousand francs to be paid at compound interest. The sums of twelve and one-half millions are thus distributed; – first, eight millions five hundred thousand among the religious communities: secondly, two millions and a half among the secular clergy: thirdly, four hundred and twenty-five thousand among the hospitals, dispensaries, and infirmaries. \n",
      " These articles are like all other articles in a newspaper: they belong to that class of reading which has no existence whatever out of that paper. What we understand by things is quite different from what an uninstructed person would understand by them – or rather (to say it in the manner of a philosopher) there is no such thing as things, but only words and phrases. It cannot be expected that those who never look behind the printed page should see any meaning in such phrases. They must take them down for what they are worth; and not waste their time in attempting to make anything out of them. \n",
      "The same process may be pursued by every other class of persons, as far as it goes; but of course it does not go far enough to satisfy the curious – or even the vulgar: to whom there is no end of such nonsense, unless they are able to produce some real matter out of them. If they do manage this, their discoveries become the property of the public, and no longer concern those who have occasion for them only in connection with newspapers. Thus, a certain person (as Mr Tite Barnacle tells me) took up his pen one fine day to write an article upon 'The Irish Question'; but having got into his head that it was a very difficult question indeed, he could not possibly bring himself to do justice either to the question or to any of those who were at all interested in it; and so, after having given it up as unmanageable, he fell back upon the newspapers, and there found a subject which he had plenty of material for. The public were delighted with him because he gave them a very different set of Irishmen from what they had ever seen before – a sort of race of ferocious monsters; and when he afterwards came to Ireland they had quite made up their minds that nothing but the strong arm of the law would be able to keep order, even if peace were declared. When Mr Tite Barnacle's paper was read over at the office where I am employed as a clerk (having been in that department from my earliest youth), a great noise arose; and all sorts of wild opinions were given by the editors about how such and such articles should be treated when they come next. But this is merely what one might expect from persons who are always looking at their own words instead of at things, and can never understand that any sort of meaning must lie somewhere else than in a column of print – or at least between two lines of it: for, if the matter were not there at all, how could we ever understand what was meant by the writer? But it seems to me that some writers are as little concerned with matters themselves as with words when they come to their task. I have read many articles in the newspaper which have had a great deal of power and strength about them; but yet they might equally well be spoken, or even thought.\n",
      "When one looks at things with one's own eyes, there is no room for any doubt whatever. We may see clearly what sort of creatures we are dealing with: it makes very little matter that we are not allowed to see their faces and can only guess their character from what they say. It has often been observed by writers in the papers that the English and Scotch are very much alike; but this is only a vague general idea, which it would be easy to refute if one had occasion. I have travelled over both countries more than once – though not for the purpose of studying them from an ethnographical point of view – and when I have met English people I could always tell who they were at sight; but if the same thing happened in a Scottish place,\n",
      "llama_print_timings:        load time =   13046.79 ms\n",
      "llama_print_timings:      sample time =     561.98 ms /  1024 runs   (    0.55 ms per token,  1822.14 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2128.74 ms /   494 tokens (    4.31 ms per token,   232.06 tokens per second)\n",
      "llama_print_timings:        eval time =   28539.57 ms /  1023 runs   (   27.90 ms per token,    35.84 tokens per second)\n",
      "llama_print_timings:       total time =   31540.50 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "33860261-8154-4b7c-bdec-cccc2626ba86",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326075\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors she entertained themselves much with the preparing and drawing up of maps of heavenly Jerusalem, and of plans of conduct prescribed for the kings of the world in the way to dwell there. The English even, in their clergy, who were soon after so alert on pretended communications from the skies, had not been slow (what wonder?) to give out alarming speculations of that kind, concerning the destination in the other world of the Southern Confederacy. \n",
      "But the London season was drawing to a close – the happy inhabitants were combining the advantages of an early rise and a late dinner. Soon Christmas-day would be here; soon it would be a Happy New Year again: or, rather, it would be Happy New Years always, one after another, forever and ever. \n",
      "The Dedlocks, being on their way back to Windsor, had been compelled by the inclemency of the season to stay at an inn on Hampstead Heath; and so had Mr. Tulkinghorn. In pursuance of his inquiries he was up with the lark and had already made a start, when it seemed to Mr. Tulkinghorn that he heard, outside the window, a horse's footsteps sounding on the stony road beneath his window. It could not be John Jarndyce; for, though John would have taken the very earliest opportunity of seeing Mr. Tulkinghorn and of asking after poor Richard, yet he was now in town and Mr. Tulkinghorn's inn stood on Hampstead Heath. The sound was as if it came from a distance, and Mr. Tulkinghorn thought it might be somebody riding by the house. But the horse seemed to stop outside, so that Mr. Tulkinghorn could hear no more of him till the animal raised his hoofs again. This time they were nearer and it seemed to Mr. Tulkinghorn that there was some person in the saddle.\n",
      "It may be remembered what a long ride Mr. Vholes had taken when he called upon the family, and how Mr. Tulkinghorn had looked after him from his window, and had seen him dismount in the road before his gate, as if he had been coming home all the time. This was more than two months ago; it may be remembered that Mr. Vholes was then, according to Mrs. Rouncewell's description of him, a thin man, and that now –\n",
      "Now, if Mr. Tulkinghorn happened to be standing by his window looking out on to the road, he would see at a glance what sort of a person it was who rode up. It could hardly be anybody else but Richard's lawyer; and yet there was something in this very man, the horse, and the manner of riding that did not seem quite to accord with Mr. Vholes as he had known him before. The rider appeared to be a man of much lesser dimensions than Vholes: the horse was a small animal with some shaggy hair on its shoulders; Vholes's coat and hat were out of fashion; and in the whole person there seemed an air of carelessness and slovenliness which was quite foreign to Mr. Vholes. The lawyer did not stop his horse at Mrs. Rouncewell's gate, but drove straight up to Mr. Tulkinghorn's window with a clatter of hoofs that were startling in the still morning air.\n",
      "'Good-morning, sir,' said this strange creature as he drew rein and dismounted, 'I hope you are well. My name is Gridley: I am Richard's lawyer.'\n",
      "Mr. Tulkinghorn stared at him in some astonishment. The lawyer was young – not much more than half Mr. Gridley's age; but it could scarcely be said that he looked like a young man; and there was nothing of the clerical look or dress about him, but rather of that wild uncouth appearance which belongs to a professional gambler in the streets of London.\n",
      "The lawyer turned his horse out into the yard, and then came up close to Mr. Tulkinghorn's window. 'You don't know me yet, sir,' said he with a laugh; 'but you are going to – eh?'\n",
      "'I do not understand what you mean by your words \"yet\" and \"going to\",' said Mr. Tulkinghorn. He had not been aware that his visitor was so familiar with the phraseology of a street-gambler, but it now came into his mind that there were some points in which his manner might have reminded him of some old acquaintance whom he could scarcely recall at first, and to whom he should be inclined to say, 'If you have\n",
      "llama_print_timings:        load time =    7145.07 ms\n",
      "llama_print_timings:      sample time =     570.98 ms /  1024 runs   (    0.56 ms per token,  1793.40 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2140.25 ms /   494 tokens (    4.33 ms per token,   230.81 tokens per second)\n",
      "llama_print_timings:        eval time =   28550.49 ms /  1023 runs   (   27.91 ms per token,    35.83 tokens per second)\n",
      "llama_print_timings:       total time =   31570.20 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "05e70274-169b-4be2-9f7c-553eb99d6361",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326115\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors she entertained herself, meanwhile, in the eighteenth century fashion, with long words and a little singing no man understood. \n",
      " In such circumstances, neither the wisest nor the stupidest of their leaders failed to perceive the necessity - the imperativeness even - of some change for the better in the promising aspect of public affairs: that is, they did not fail to perceive all this but were agreeing to disagree as to the remedy. Yet, with that hopefulness which ever mingles in peculiar degree with the darkest time of year, that epoch might be considered as ushering in a season of intellectual renovation and fast-coming prosperity. The mass of the British nation - the rightful lords of the soil - had then attained their long looked for freedom, from an hereditary despotism and an autocratic government: they thought they might venture to assert their rights as freeborn subjects; and they determined at last that ENGLAND should be governed according to the will of ENGLAND'S PEOPLE. This was a new departure - this was at least to be a novel experiment!\n",
      "The French Revolution was in progress - and the event of the Eighth Day of Thermidor, An II [26th July 1795], had established an unchangeable era in the destinies of France. It was a period big with events too important for even Napoleon himself to overlook; and his whole policy turned upon its results. He never could forget how he had been left behind by it: and he felt it imperatively necessary that a great battle should be fought, somewhere near Paris - in or around the Department of Seine-et-Oise - before any longer delaying his return home to join his fellow-citizens at the feast of victory.\n",
      "NAPOLEON BONAPARTE was then Commodore-in-Chief of all the French forces on land and sea: but he had been in that office for more than five years, during which period, according to his own account, not one day - certainly no night or nights! - did he ever lie down without a headache; and at last this continual state of tension was telling seriously upon his health. It is said that his physician, Baron Larrey, found it necessary in order to cure him of his ailments, to remove him altogether from the theatre of warfare. This was accordingly done; he was carried on board one of his own frigates and sent to join Admiral Bruix at A Coruña (in Spain): and thus, though but little more than thirty years old, he became in 1803 a retired naval officer!\n",
      "It is not my business to discuss the various theories that have been propounded concerning Napoleon Bonaparte - of which I am inclined myself to regard only two as being at all worth considering: and these are only his early history in Corsica (of which, for instance, his second brother Lucien wrote a book) and his first expedition across Europe into Italy. All other narratives either appear to me to be founded upon some preconceived theory or upon the personal gossip of the author: - but it may not be out of place to say that the latter are quite as likely, perhaps even more likely in most instances, than the former to be false - and also to remind the reader that Napoleon himself did not deny being the son of a French official at Corsica who had been married there before he returned to France. In all these circumstances I shall never feel inclined to consider his character or even his acts as being either so great or so good.\n",
      "I have now, however, come upon a question which, though it may be of little importance to myself (and therefore not worth investigating), is yet one of considerable interest to many others - and that is the question of what is meant by \"The First Consul?\" As it appears to me very difficult in any way whatever to arrive at an exact understanding as to whether he was or was not a King, I shall, before giving my own opinion upon the matter, set forth as much information concerning it as I can gather from the books and pamphlets that have been published by those who were actually present during his reign.\n",
      "As far as I am able to judge the question may be divided into two parts: firstly, whether he was King of France before the coronation (or if not of France, then of Italy): secondly, what title should be given to him after his crowning? It is clear that no one who had been elected to a position would foresee so great and sudden an honour as the one conferred upon Napoleon by the Senate - but it would also appear to me that there can hardly be any doubt as to the answer of\n",
      "llama_print_timings:        load time =    6866.28 ms\n",
      "llama_print_timings:      sample time =     572.75 ms /  1024 runs   (    0.56 ms per token,  1787.85 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2131.31 ms /   494 tokens (    4.31 ms per token,   231.78 tokens per second)\n",
      "llama_print_timings:        eval time =   28555.28 ms /  1023 runs   (   27.91 ms per token,    35.83 tokens per second)\n",
      "llama_print_timings:       total time =   31568.82 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c2c1009-0642-4be0-8fee-f22a3e8bc23a",
   "metadata": {},
   "source": [
    "### 13B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "134a3c7b-30e2-460c-bfcb-ccb51c090797",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326155\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its noble architect, Monsieur François A. F. Arouet, called Voltaire, and kept in the straight path by Jean Jacques Rousseau, that great rainbow in the sky which we call the French monarchy extended itself over half the globe. Doubtless the good people who lived under it were grateful to it for their warm succour and protection; and glad to have so vast a system of spiritual suggestion always at hand and ready to shower down its daily portion of heavenly comforts and celestial benefits on any head that might require them.\n",
      "France had taken a new step in the career of progress. Of old, it had been the custom among the French to set apart one day in every week for repose from labour, and devotion to the service of God; but, at length, some industrious soul had discovered that it was highly desirable that the same end should be attained in less than a single day, and only two or three hours spent on such occasions. Accordingly, the good old custom was abolished by law: and, instead of resting one whole day out of seven, Frenchmen were permitted to get through the whole affair in no more than half an hour! Now, this is not at all a bad observation upon the subject; for it must be granted that it is possible, and even probable, that a very great deal of hurry-skurrying goes to the making up a tolerable performance for such short space as I have mentioned: whereas, if the period were seven hours or seven days, there might be a chance for a little more leisurely preparation, and a rather better execution of the work. When we consider what a vast deal of sheer ignorance (for it is plain that information of any kind was quite out of the question) and downright impudence in presumption went to the composition of this half-hour whirlwind, it will hardly be supposed that much time or many thoughts were wasted on it; for it was made up entirely of extempore flourishes by a set of fellows who called themselves ballet dancers. The very word\n",
      "\"Ballet\" is a corruption of the Italian \"ballerino,\" signifying a dancing master or teacher, and from this source, in course of time, have sprung all those precious specimens of modern art which nowadays astonish us under the names of _pantomime_ dancers; or\n",
      "\"pantomimists;\" as they are termed by way of distinguishing them from other people. A good deal of the same sort of stuff was practised in those days, and on the stage itself; though at that time it was called \"drollery.\" There was an actor who made a great noise for some years together by representing \"a dancing bear;\" and there were likewise two or three buffoon dancers who went about, dressed sometimes as \"Turkish Spahis,\" and sometimes as \"Hussars;\" in the latter character they wore, I recollect, immense mustaches; but their best business was when they got into the costume of a German Band. At this time, too, there was another dancing actor--one who has since obtained an honorable retirement from the stage in the capacity of master to the late Princess Amelia's little children; I mean \"Williams.\" He was at that time very great on the boards; and was called by way of distinguishing him as\n",
      "\"_Little_ Williams.\" The latter name, however, he has long since dropped. But there is one more particular individual whom I must be permitted to mention in this place. This was \"the celebrated _Jack Bannister_:\" who flourished on the London boards about the period of which I am now writing; and for many years after that.\n",
      "\n",
      "But, returning to the subject with which I began--and speaking merely as a critic, or rather as an amateur critic--the truth is, that if one could have had his own way, there would not only have been no \"Little Williams,\" but also no \"Jack Bannister.\" And this for two reasons. The first and most important of all, being, that theatrical representation has no business with burlesque.\n",
      "\n",
      "In the second place, therefore--and in reference to this one point at least, I will venture to say that it is very much better than people have any idea of; or even suppose possible. The case is thus: It does not follow, because an actor can speak and act in his proper person--which indeed is a very easy thing to do, when the character requires nothing more than the ordinary actions of common life, and the language which men speak at their own doors, or in the streets, and other public places,--it does not therefore by any means follow that he should be able\n",
      "llama_print_timings:        load time =    6238.82 ms\n",
      "llama_print_timings:      sample time =     576.40 ms /  1024 runs   (    0.56 ms per token,  1776.54 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3350.56 ms /   494 tokens (    6.78 ms per token,   147.44 tokens per second)\n",
      "llama_print_timings:        eval time =   29692.50 ms /  1023 runs   (   29.02 ms per token,    34.45 tokens per second)\n",
      "llama_print_timings:       total time =   33932.02 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "aad047cf-d4d1-4e14-937f-1dd1e456a11b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326197\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she was fascinated by rays from Olympus, and followed in the paths of former iniquities, and made a spritely business of being outrageous. England, under the guidance of her Christian pastors, was insensibly becoming outrages in itself, to which all France bowed down in vast obsequiousness. \n",
      " Madame Tussaud had just set up her wax-work in London; the shops were full of three-cornered hats and military cockades: Brighton was prohibited to ladies, on pain of being turned out of a regiment, if they presumed not so much as to look through the windows of the Royal Pavilion. M. Talleyrand, lately Prime Minister, had returned to France and died; his low-necked wits and double-faced politics were, however, still in vogue. Mrs. Fitzherbert too was back in England; but Mrs. Jordan – for whose sake that accomplished lady made her way across half the world, and whose name remains on the border of the Bayeux Tapestry – was still at Naples, under the auspices of Madame Campan, sister-in-law to Talleyrand, wife (with a difference) of M. Fouché, Minister for Foreign Affairs, and mamma of Madame Bertrand.\n",
      "The Empress Josephine had died in 1810; but her memory survived in the Imperial Harem at Compiègne – where, not many years afterwards, little Charles Louis Bonaparte was to be born – and in all the Imperial palaces from the Tuileries to the Elysée.\n",
      "The French nation had been formally divorced from Josephine; and it was announced that a new marriage was projected for Napoleon, which would have the effect of restoring him in the good graces of Pope Pius VII. But, even while these overtures were being made to Rome, an unexpected marriage occurred in the French Imperial Household, whereby one of the stepson sons of Napoleon became the husband of one of his stepdaughters – a marriage which, under existing circumstances, seemed likely to have about as much influence on the position of Napoleon and his family in Italy and Austria as if he had married his own daughter.\n",
      "Thus, early in May, 1809, Eugène Beauharnais was married, at Vienna, to Princess Hortense de Beauharnais: the very youngest child – then about fourteen years old – of the first marriage of Josephine (afterwards Empress of France); who had been adopted by Napoleon, and brought up by him as a daughter. The new Emperor of Austria was among those present at this strange marriage: having arrived on that occasion from the Rhine – where he was superintending his German troops – to take part in the nuptials of his cousin-in-law Eugène Beauharnais and his step-mother's step-daughter Hortense de Beauharnais. The marriage is remarkable for this: that its celebration at Vienna was entirely dependent on a certain fact, which nobody had any knowledge of – till after it was celebrated: namely, the Emperor's having, some days beforehand, granted Eugène Beauharnais the Grand Duchy of Frankfort (which had been wrested from Baden, and had been since given to Bavaria), and that Grand Duchy of Frankfort being the fief or hereditament which was assigned as a marriage-portion by Napoleon to his stepdaughter Hortense. It is worthy of remark, that there existed no such Grand Duchy at this time; and that, when it came to be arranged, a small Principality was found available, and named Frankfort: being in fact only the small old territory called Waldeck which lies to the eastward of that of Brunswick. The title of Grand Duke, given to the new holder by Napoleon, was one thing; but the new dignity had its own peculiarities of situation, to be ascertained by his Imperial Majesty: which in fact were nothing more than that Frankfort was to have a Government; and that he was not only to be the sole personage there, but also the first one to whom the title of Grand Duke was due.\n",
      "The Grand Duchy of Waldeck (or Frankfort) was then made ready for its new owner: with such equipments as it had, in the way of an Army and Navy; so that he could be at once ready for battle or invasion anywhere. There are very few things more instructive to a young mind than watching over such an eventful transaction as this is; how it all came about in one way, and then went along towards the other: and especially with respect to the matter\n",
      "llama_print_timings:        load time =    7077.34 ms\n",
      "llama_print_timings:      sample time =     570.28 ms /  1024 runs   (    0.56 ms per token,  1795.61 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3359.12 ms /   494 tokens (    6.80 ms per token,   147.06 tokens per second)\n",
      "llama_print_timings:        eval time =   29838.38 ms /  1023 runs   (   29.17 ms per token,    34.28 tokens per second)\n",
      "llama_print_timings:       total time =   34082.37 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb28eb3a-3cee-45c0-b25b-ad68ad8bdcf5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326240\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its Christian priesthood, Religion dispensed, like misnamed Polygamy, harmless consolation to innumerable men and women; counted worthy of that mildest word (so far from being reproachful!) which modern blasphemers have presumed to usurp upon it; and, when the last trumpet shall sound, will move on the bruised surface of the globe, and be extinguished nowhere.\n",
      "Men, my brothers, men the workers, ever reaping something new: that which they have done to-day, but always different; for, to-morrow is a new day; and we are grateful to it, not old or young, for its first light forests, where no track of men appears; its resounding streams, its significant mountains. We are presently invited to meet a fresh revelation from the same old guest and entertainer, Time. You saw yesternight some natural thing enlarged to an unnatural size, as by accident—a crab like a spectre, an eel like a snake, an ant like a giant; or, last night, some spectral deformity, too slight for its own outlines, that gave it the semblance of being very large. To-day, we shall have the portrait of some elderly man who yesterday was young: to whom, in the mean time, an impersonation of his former self has been vouchsafed, which he by no means understands: but we know what it is like, for it was shown us last night. The ship with all hands at the pumps—that was last night's revelation; and a marvellous one! What made it so pregnant with fate and time, was that when our minds were all diverted towards the captain and officers on deck by some ado below; when we suddenly heard a scuffle of many feet on the forecastle, and in a moment after saw a multitude of shadowy dumb-bells descending from aloft—one over another—like magic stairs; and last, two figures; one of which, in its fall, so hung upon the other, that he seemed to drag it down with him. This is but another shape which Time and Fate (as we don't doubt that they have some disguise for every fancy) assume, that so they may work on us as goblins, and scarcely be discerned; a shape which we had not contemplated before, and the sight of which made us feel that there was something very strange in human nature. For what are all these but so many mortal accidents, of no account to any one else, that must be borne with by him who is saddled with them; for him they are as hereditaments, indelible and indefeasible; as the physical marks in his body—nay, we might say in his soul—of all the external influences, pleasures, pains, griefs, and joys he has known, resistless and inevitable, like storms, or change of season: they are in his frame, a part of himself. All the world's people have these mortal accidents; but no two men have the same. A man must bear his own accidents, without whining.\n",
      "In order to give full effect to what we now had it in us to say, it was necessary that the audience should be silent, and should not only feel, but understand, how much the speaker felt what he said; that is, he would have us to know why, as a part of this man's own nature, so many things were a part of his life.\n",
      "Therefore we stopped awhile. We knew the value of silence. But there was another thing, and it was in order to make our own feelings more sensible and definite, that we made our audience wait. For after all, the feeling of what one is about to say does not come on us at once; but slowly grows up, as a thought comes from within, not from without. And the same with this man. It was not that he had lost his faith in anything; no, that he could not have lost: for how can a man lose the only thing that he has to believe in? But it is so with all men: when their soul's house of life becomes a little dilapidated in them; and they feel themselves broken up into fragments; and their old pleasures are dropping off of them; and they know not what to call upon for comfort, but only to look back on the past, when every thing was bright, fresh, new, and full of hope;—then it is that this man's thoughts began to work in him: and he had to see wherein consisted his own\n",
      "llama_print_timings:        load time =    3494.24 ms\n",
      "llama_print_timings:      sample time =     571.61 ms /  1024 runs   (    0.56 ms per token,  1791.43 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3350.36 ms /   494 tokens (    6.78 ms per token,   147.45 tokens per second)\n",
      "llama_print_timings:        eval time =   30381.28 ms /  1023 runs   (   29.70 ms per token,    33.67 tokens per second)\n",
      "llama_print_timings:       total time =   34616.14 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d1552a-9361-4433-aa57-b616ecd13dee",
   "metadata": {},
   "source": [
    "### 13B f16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "24e9abc3-f9fc-4faa-9d60-d8971e686e69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326280\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 24.24 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  312.64 MiB\n",
      "llm_load_tensors: VRAM used           = 24514.08 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 24989.09 MiB (model: 24514.08 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she entertained herself, besides, with such humane achievements as sentencing a youth to have his hands cut off, his tongue torn out with pincers, and his body burned alive, because he had not kneeled down in the rain to do honour to a dirty procession of monks which passed within his view, at a distance of some fifty yards. It is likely that, rooted in the woods of America, she saw nothing of this monster, but her prophet in the nineteenth century saw it all clearly. \n",
      " England had commenced to follow the example of her transatlantic descendants in asserting the right of self-government, and as a natural consequence of being the sole defender of her own highways, was preparing an army with which to enforce that right. At such a time, Mr. Lorry must have been mad indeed, if he yielded to any fear of the world in which he lived, and of the state into which it was tending fast.\n",
      "Yet, Mr. Jarvis Lorry had been known to yield to fear.\n",
      "Chapter I – The Preparation\n",
      "A SHABBY figure slunk into the Nuns Head taproom. It was a murky evening, with a fog creeping in off the Thames and rolling round the dim cathedral of chimney-stacks that encompassed the neighbourhood of St. George the Martyr. The fugitive stole to an obscure table in an obscure corner near the door of the taproom; and there he sat, hidden by the shadow of a screen, and whispered fiercely to himself.\n",
      "‘The winds will blow! The winds will blow!’ he muttered, over and over again: ‘and then, where shall they drive the good ship Nell, and who shall steer her? Windy weather coming on; windy weather coming on.’\n",
      "He was a ragged figure as he sat there. The shreds of an old sail, flapping in a strong sea wind, would have been no bad likeness for his faded blue jacket. His knees peeped out through the rents, like stumps of masts; and the trousers hanging at his legs were holed and rotten like a ruined spar after much water. In that wild year of our Lord 1792, he was not without cause to be in a bad way—but had his head so far into an old woman’s apron pocket, who sat on the opposite side of the fireplace and pondered over her tea-leaves, that there would have been danger in a glance from him.\n",
      "Therefore, he was not observed; but he continued to mutter ‘The winds will blow! The winds will blow!’ with such feverish iteration and vehemence that it might be doubted whether the air or he were hotter. As for his shoes, they had been at sea long enough for their own good: they were never heard of afterwards—and would probably have sailed down to a watery grave, like their old comrade before described; but for being held on at all points by various scraps of ribbon and strips of cloth.\n",
      "‘Windy weather coming on,’ he repeated; ‘windy weather coming on.’\n",
      "There were plenty of reasons why the weather should have been blowing hard upon that figure, sitting in the shade of the old screen. It had once been a sailor—it was not ashamed to show how worn and weather-worn it was, for such tokens were badges of pride among them; but the seas had grown cold or rough, or both, to him, and he was landed here at last. The place, though very quiet in the evening, when there was no sound but that which it derived from London, where its river flowed—and then only an echo as of the great heart beating at a distance; and the lighted windows making eyes for all the darkness beyond them—was by day a place of traffic. Merchants, sailors, poor creatures in rags (who had no doubt been sailors once), idle people who never could find their way to sea if they were given ships with sails upon their backs—all sorts and conditions came down to that spot, to sit before the door of a public-house there; which was not at all an unreasonable thing, for it was known by a very fine and gallant sign of three golden balls. There, from morning to night, they sat and gossiped about the sea.\n",
      "With these gossips the old sailor—if he were one, which seemed probable enough—had now grown quite a favourite. He came\n",
      "llama_print_timings:        load time =   25168.84 ms\n",
      "llama_print_timings:      sample time =     568.93 ms /  1024 runs   (    0.56 ms per token,  1799.87 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3406.77 ms /   494 tokens (    6.90 ms per token,   145.01 tokens per second)\n",
      "llama_print_timings:        eval time =   40677.09 ms /  1023 runs   (   39.76 ms per token,    25.15 tokens per second)\n",
      "llama_print_timings:       total time =   44966.94 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "acba2193-558f-4ec4-a8bf-ac9610e70f23",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326351\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 24.24 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  312.64 MiB\n",
      "llm_load_tensors: VRAM used           = 24514.08 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 24989.09 MiB (model: 24514.08 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its chief spirit, the country danced at his pleasure; and those who were exhausted rested.\n",
      "—A Tale of Two Cities  \n",
      "CHAPTER I\n",
      "HUNDREDS of houses, costly and sumptuous, constructed after various fanciful plans, stood huddled together on the sloping shore of Lake Leman. Some had Turkish domes, others gothic towers, many were built in the fashion of Italian palazzi: all had ornamented fronts. In the midst of these beauties rose a majestic terrace, collectively called St. Peter's, and considered as forming the most splendid mansion in Geneva.\n",
      "The rooms of this house were distributed over three floors, the highest of which consisted only of a gallery surrounded by balcony, whence a fine view of the lake, and of Mont Blanc, might be enjoyed.\n",
      "At the hour when the day's work is over, and people are pouring out of offices and warehouses into the streets, St. Peter's presented an appearance which was as busy and as cheerful as any other place in Geneva. It was a great receiving house—not for strangers only but for the whole city. The hall door was almost invariably open to let in visitors; and if one came early, he might see a great family sitting round the long table, engaged in a merry supper of which there were always at least two services every day.\n",
      "As soon as ever there appeared any likelihood that the entertainment might come to an end, the servants had orders to close the hall door, and to lock it till the next meal; after which time visitors were received as freely as before. In spite of this frequent interruption, St. Peter's was almost always full, because it was a place of business as well as pleasure.\n",
      "A few minutes before seven o'clock, on a fine evening in July, one such visitor rang the hall door bell. He waited for some time without getting an answer. Then he rang again, and after waiting some more, went away; but he did not go far enough to escape observation. He had hardly turned the corner of the house when the hall door opened and a servant came forth with a candle in his hand, which he placed against the wainscot as a guide for some one who seemed to be walking within, towards that very door. The visitor then returned, tapped gently, and being asked what he wanted, said he had come by appointment to see one of the family,—a young lady,—Miss Manette.\n",
      "The servant showed him upstairs into a large room lighted up, where several people were sitting. The master of the house advanced towards him: a little man with an agreeable countenance, in whom it was difficult for a stranger to recognise the original of the large print and bizarre style which accompanied the announcement that he was 'the celebrated M. Darnay, just arrived from France.' He greeted his visitor with much civility, and presented Miss Manette, saying:\n",
      "'She is one of the family whom you expected to see.'\n",
      "She was a young lady whose age seemed to be in its teens, though her appearance, which was remarkably precocious and decided, would have satisfied us that she must have been years older. There was no beauty in her face, nor any sign of it in her figure; yet she was so pretty that nobody who saw her ever forgot her—I never did from the time when I first saw her.\n",
      "She had a dark complexion (almost brown, like a gipsy) and straight black hair; she was tall and slender, with an exquisite figure; her face, naturally pale and pleasant-looking, was rendered unnaturally pale and unpleasant in appearance by want of colour. But there are some characters which always please, whatever they do—some faces which have a kind attraction for you even when distorted into ugliness—and so there is something about this young lady which, despite her weather-beaten aspect, touched me as soon as I first saw her. It may have been in its own way the same kind of attraction that the foulest monsters in fiction have for us: I don't know; but I know it was a real attraction. She was simply dressed, and had nothing pretty or elaborate about her save that she was very neatly dressed.\n",
      "Her father (her mother having been dead many years) said to his visitor in the course of their talk—referring to her:\n",
      "'She has always been the same since her illness.'\n",
      "The young lady overheard, and at first her face flushed; but she controlled it, and had a quiet and decided manner as she spoke.\n",
      "\n",
      "llama_print_timings:        load time =   27168.19 ms\n",
      "llama_print_timings:      sample time =     578.16 ms /  1024 runs   (    0.56 ms per token,  1771.12 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3414.71 ms /   494 tokens (    6.91 ms per token,   144.67 tokens per second)\n",
      "llama_print_timings:        eval time =   40595.89 ms /  1023 runs   (   39.68 ms per token,    25.20 tokens per second)\n",
      "llama_print_timings:       total time =   44903.53 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d998b1f9-2783-4389-bfc9-bb2999d8a113",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326425\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 24.24 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  312.64 MiB\n",
      "llm_load_tensors: VRAM used           = 24514.08 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 24989.09 MiB (model: 24514.08 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its Christian pastors, it perfected itself in Holy-Roman-and-Apostolic superstition, exterminated the jews, imprisoned the protestants, conveyed the finest of her plunder to the Vatican, and carried the rest of her plunder to-the gun-maker's, the bullion-dealer's, the banker's, and the private speculator's. Then, the people rested. \n",
      " England, conscious of being the leader in civilisation, looked with a tender regard at the struggles of France, and hoped to be able to help her in those struggles, when they should assume a more favourable shape. England could not see with patience France going to rack and ruin under a government of men who knew what they were about. \n",
      " In point of fact, these gentlemen seldom knew any more than what they were saying; in other words, they were unconscious whereof they affirmed. The English Court looked with a troubled countenance on the French Terror. It was glad to make any observation that tended to turn men's minds from it. \n",
      " One day, Mr. Pitt asked his colleagues if they had seen the speckled band which was playing at the Egyptian Hall? No. They had not observed it. Mr. Pitt advised them to go and see it, because it was one of those things which were too ridiculous to be believed, but which everybody ought to see. \n",
      " England used this precaution with her neighbours as well as herself. She looked with a troubled countenance on the French Terror; she advised foreigners (as well as her own citizens) to go and see it, because it was one of those things which were too ridiculous to be believed, but which everybody ought to see. \n",
      " England saw with astonishment that these measures produced no effect in France, and that the Terror went on as if all men's opinions concerning it were unchanged. England called to her aid her old friend Austria; she lent an eager ear to Russia; she patted Italy on the back; but the Terror only skirmished a little, and then sat down to its dinner again. \n",
      " At last when it had dined, it went out for a walk, and met an Englishman in his shirt-sleeves, who called out in a jocose manner, \"Why don't you wash yourself and come home?\" But the Terror only looked at him, and took no notice. \n",
      " At this, says England, I am surprised! Upon my word I begin to think that all men are fools! \n",
      "\n",
      "\"I can't make it out!\" said John Bull, tapping his leg with a cane and glaring angrily on the ground: \"it is too bad. All men appear determined to do nothing. The French nation dines, walks out, kills people; then comes home to dinner again, as if there was nothing in the world worth taking notice of but a good dinner and a bit o' cleanliness now and then. The other nations skirmish and make treaties, as if they were playing at battledore and shuttlecock; and all seem determined not to mind what they are about!\" \n",
      "\n",
      "\"Upon my word,\" says John Bull, \"I begin to think that the whole world is in a conspiracy against me.\" \n",
      "\n",
      "The reader will see that England was beginning to feel very much annoyed, when Fortune favoured her with the opportunity of assisting in what is known by the name of the Battle of Waterloo. This battle, says John Bull, is a decisive one, and has been fought upon the 18th of June. The French have been completely defeated, the army dispersed, and Bonaparte taken prisoner: which he was to have expected, as he had made no preparation for defence on his part. \n",
      "\n",
      "The English army (says John Bull) behaved remarkably well throughout; but, in point of fact, I do not think they did half so well as the Prussians and Dutch, who were under my command on that occasion. These people are an unaccountable sort of beings; they have no hearts in them! \n",
      "\n",
      "I have been very ill for a week or two after that affair (says John Bull) and could not get out to see the King come back again. But I did send my card. You see what is written upon it, sir:—\"John Bull.\" That's a name; ain't it? \n",
      "\n",
      "Now, the King came home in triumph on the 8th of July: and immediately after he came back, the\n",
      "llama_print_timings:        load time =   19716.49 ms\n",
      "llama_print_timings:      sample time =     563.32 ms /  1024 runs   (    0.55 ms per token,  1817.79 tokens per second)\n",
      "llama_print_timings: prompt eval time =    3413.23 ms /   494 tokens (    6.91 ms per token,   144.73 tokens per second)\n",
      "llama_print_timings:        eval time =   40583.87 ms /  1023 runs   (   39.67 ms per token,    25.21 tokens per second)\n",
      "llama_print_timings:       total time =   44870.13 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04edde69-2bb1-4cd2-b0af-e265f93f128b",
   "metadata": {},
   "source": [
    "### 70B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0aac592f-7a5c-41b1-88f5-f4d3157af8b1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326492\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type q4_0:  561 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 8192\n",
      "llm_load_print_meta: n_head           = 64\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 80\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 8\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 28672\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 70B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 68.98 B\n",
      "llm_load_print_meta: model size       = 36.20 GiB (4.51 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.28 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  140.90 MiB\n",
      "llm_load_tensors: VRAM used           = 36930.11 MiB\n",
      "llm_load_tensors: offloading 80 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 81/81 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 160.00 MB\n",
      "llama_new_context_with_model: KV self size  =  160.00 MiB, K (f16):   80.00 MiB, V (f16):   80.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 1684/1684\n",
      "llama_new_context_with_model: compute buffer total size = 148.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 145.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 37235.11 MiB (model: 36930.11 MiB, context: 305.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she entertained herself, besides, with such humane achievements as sentencing a youth to have his hands cut off, his tongue torn out with pincers, and his body burned alive, because he had not kneeled down in the rain to do honour to a dirty procession of monks which passed within his view, at a distance of some fifty or sixty yards. It is likely enough that, rooted in woods of La Vendée, where the Druidical religion had lingered longest, the resistance to the Republican movement had revived no small store of old dissensions and hereditary local animosities, which rendered the opposition much more obstinate, protracted, and bloody, than it might otherwise have been. But, as to this, there were bloody groundless forgeries then, engendering fierce and cruel passions, as we have seen in our own time; and there was a Queen of France (one of the most remarkable women ever born), whose hair they say, turned white in a single night through grief at the horrors she beheld. \n",
      "It is likely enough that it was the obscure consciousness of having set up something false upon their countryside, which caused them to be so violent and wrathful against him who shed light upon it from the first, as they had reason to think this light would shine. But, there was another way in which the child could be restored, which did not come into such direct collision with any of the laws; and that other mode was a certain scheme for his restoration, devised by Doctor Manette. It included much; perhaps everything.\n",
      "The son of the Prince of Wales had died two years previously of smallpox brought on from vaccination, and it was this fact which led to Jenner's famous experiment.\n",
      "It is likely enough that in the rough outhouses of some tillers of the heavy lands adjacent to Paris, where agriculture was still languidly pursued, he found shelter, and concealed himself through the day. The French had now undoubtedly begun their retreat, and the Cossacks finding the woods clear of them, descended into the valley in the hope of obtaining food and quarters for the night.\n",
      "A certain number of people had discovered a means of being fed, without discovering any other means of earning money. With these the old gentleman laid himself out to be extremely facetious and humorous; making jests at his own expense, telling stories against himself, and courting ridicule in every possible manner.\n",
      "He was a very clever man, and soon learned, as he expressed it, 'to make a noise'.\n",
      "The French had now undoubtedly begun their retreat, and the Cossacks finding the woods clear of them, descended into the valley in the hope of obtaining food and quarters for the night. At this time one of Napoleon's most experienced generals was in command of the town; he represented that it would be impossible to resist an assault on the part of numerous and fresh troops, and advised surrender. He was accused afterwards of wanting to betray his master into the hands of the Cossacks, but it is more likely that he simply wished to save the lives of so many brave fellows by a capitulation, though there are instances enough of French soldiers being shot down even after they had laid down their arms, and when no resistance could be apprehended from them.\n",
      "He was accused afterwards of wanting to betray his master into the hands of the Cossacks, but it is more likely that he simply wished to save the lives of so many brave fellows by a capitulation, though there are instances enough of French soldiers being shot down even after they had laid down their arms, and when no resistance could be apprehended from them.\n",
      "Those who have seen me in public know how well I play my part; I am naturally grave and austere in private. This, then, is what I am. But those who are not acquainted with the man, take him to be stern, haughty, hard-hearted, unsociable, disagreeable: and so he must be taken, as he is really all this.\n",
      "I have been asked to give a few words of advice to young men who may be thinking about entering a monastery; but there are not very many such young men in the world at present, I think. However, to those who do exist I will address myself: and I say first of all, try to realize what it is that you are intending to do, before you take any irrevocable step.\n",
      "Try to realize what it is that you are intending to do, before you take any irrevocable step. You are not going among str\n",
      "llama_print_timings:        load time =   38024.22 ms\n",
      "llama_print_timings:      sample time =     565.91 ms /  1024 runs   (    0.55 ms per token,  1809.47 tokens per second)\n",
      "llama_print_timings: prompt eval time =   11973.18 ms /   494 tokens (   24.24 ms per token,    41.26 tokens per second)\n",
      "llama_print_timings:        eval time =   75643.68 ms /  1023 runs   (   73.94 ms per token,    13.52 tokens per second)\n",
      "llama_print_timings:       total time =   88494.55 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/70B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "af6de14d-49ba-43a4-9ff0-6e90c5b57080",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326620\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type q4_0:  561 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 8192\n",
      "llm_load_print_meta: n_head           = 64\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 80\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 8\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 28672\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 70B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 68.98 B\n",
      "llm_load_print_meta: model size       = 36.20 GiB (4.51 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.28 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  140.90 MiB\n",
      "llm_load_tensors: VRAM used           = 36930.11 MiB\n",
      "llm_load_tensors: offloading 80 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 81/81 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 160.00 MB\n",
      "llama_new_context_with_model: KV self size  =  160.00 MiB, K (f16):   80.00 MiB, V (f16):   80.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 1684/1684\n",
      "llama_new_context_with_model: compute buffer total size = 148.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 145.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 37235.11 MiB (model: 36930.11 MiB, context: 305.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she entertained herself, besides, with such humane achievements as sentencing a youth to have his hands cut off, his tongue torn out with pincers, and his body burned alive, because he had not kneeled down in the rain to do honour to a dirty procession of monks which passed within his view, at a distance of some fifty or sixty yards. It is likely enough that, rooted in the woods of France and Norway, there were growing trees, when that sufferer was put to death, already marked by the Woodman, Fate, as coming timber for the gallows that were afterwards to hang him. Possibly in the rough outhouses of some tillers of the heavy lands adjacent to Paris, there were sheltered from the weather that very day, rude carts, bespattered with rustic mire, snuffed about by pigs, and excelled in hidden infirmity by the precarious tillage, that in less than half a century were to bear tumbrils along streets of Paris. But, that Woodman and that Fate, though they work unceasingly, work silently, and no one heard them as they went about with muffled tread: the rather, forasmuch as to entertain any suspicion that they were awake, was to be atheistical and traitorous.\n",
      "In England, there was scarcely an amount of order and protection to justify much national boasting. Daring burglaries by armed bands, and highway robberies, took place in the capital itself every night; families were publicly cautioned not to go out of town without removing their furniture to upholsterers' warehouses for security; the Highwayman in the dark was a City tradesman in the light, and, being recognised and caught, begged grace of the judge upon the score that he was a rescue prisoner from the hulks. Murders were so abundant that the interest aroused by them was next to nothing; all day long, the hangman's knife was hacking off heads in one part of the town or other, and he brought home his basket of heads in the evening almost as a matter of course. Accusations of theft were made against one lady, by another, before the King himself, sitting in Council; policemen thought it not worth their while to run after men known to be murderers. All day long, thieves slunk and sneaked about the city, with as little notice from the Police (of whom, by the way, there were not six hundred all told for the whole town) as if they had been cats; until the darkness set in, when they sprang out of their lurking places, in hundreds, and proceeded to rob, and beat, and kill. Some were known to have gone so far as to screw the people's eyes out, with a view to getting plunder from them all the more speedily.\n",
      "In this reign, the advertisement appears for the first time in England. It is like a rude forefather of the late lamented Vulture who used to eat up so many little boys and girls. He was found dead - or as it is euphemistically termed, 'deceased' - last year, on a doorstep in Clerkenwell.\n",
      "In this reign, also, the posts by the highways begin to be cut down to prevent persons being robbed as they travel. They have not all been cut down in our time yet; for we saw one standing within these few years.\n",
      "In this reign, lastly, a little poem called The Beggar's Petition, written by Sir William Davenant when he was a prisoner in the Marshalsea (for debt), is much admired and cried up. All the world are delighted with four of its most expressive verses, which run as follow:-\n",
      "A Dog starved at his Lord's gate,\n",
      "Looked up and said;\n",
      "'How mean my Lord is! He gives me\n",
      "If he 'll relieve a dog, I pray,\n",
      "Lord! teach him charity!'\n",
      "Punch would not have originated the Beggar's Petition, if he had been alive at that time. It is much more likely that Pussy, from a certain family of which we know Punch to have been a member in after ages, was the author of this composition; and if she did write it, all we can say is - shameful imposition!\n",
      "Punch is not quite satisfied about these three Georges. There were four Georges altogether; but he does not find much to laugh at in the fourth (who seems to have done a good deal more than his fair share of reign\n",
      "llama_print_timings:        load time =   40296.24 ms\n",
      "llama_print_timings:      sample time =     565.03 ms /  1024 runs   (    0.55 ms per token,  1812.29 tokens per second)\n",
      "llama_print_timings: prompt eval time =   11956.69 ms /   494 tokens (   24.20 ms per token,    41.32 tokens per second)\n",
      "llama_print_timings:        eval time =   74345.88 ms /  1023 runs   (   72.67 ms per token,    13.76 tokens per second)\n",
      "llama_print_timings:       total time =   87177.91 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/70B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "791826fb-6e09-4318-9a25-a0a9564400b4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326750\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type q4_0:  561 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 8192\n",
      "llm_load_print_meta: n_head           = 64\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 80\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 8\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 28672\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 70B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 68.98 B\n",
      "llm_load_print_meta: model size       = 36.20 GiB (4.51 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.28 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  140.90 MiB\n",
      "llm_load_tensors: VRAM used           = 36930.11 MiB\n",
      "llm_load_tensors: offloading 80 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 81/81 layers to GPU\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 160.00 MB\n",
      "llama_new_context_with_model: KV self size  =  160.00 MiB, K (f16):   80.00 MiB, V (f16):   80.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 1684/1684\n",
      "llama_new_context_with_model: compute buffer total size = 148.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 145.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 37235.11 MiB (model: 36930.11 MiB, context: 305.00 MiB)\n",
      "\n",
      "system_info: n_threads = 36 / 72 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she entertained herself, besides, with such humane achievements as sentencing a youth to have his hands cut off, his tongue torn out with pincers, and his body burned alive, because he had not kneeled down in the rain to do honour to a dirty procession of monks which passed within his view, at a distance of some fifty or sixty yards. It is likely enough that, rooted in woods of France and Norway, there were growing trees, when that sufferer was put to death, already marked by the Woodman, Fate, as coming Timber for her stake. \n",
      "Still, it was a time of great hopefulness in France, and a time of considerable constraint. The discontented temper of Louis XVI. was known to his spouse, and that neither the Dauphin nor the Dauphiness (who had become time-discoloured by age and dust) were looked upon kindly by their most Christian Majesty. Under these circumstances, the good Drouet, postmaster of Mantes (that Drouet who, as gamekeeper, had often sworn fidelity to the house of Egmont), late at night on the eighth of August, gave up to his Majesty’s soldiers a large platoon of suspicious looking people whom he had observed to pass his door in groups of from two-and-twenty to two-and-thirty. The circumstance seemed doubtful, and might not have been thought worthy of note in the year of grace 1790, but for one little item in the report of Drouet—namely, that the last words he had heard spoken by this roving armed multitude were the singularly appropriate expressions, “Liberty, Equality, Fraternity, or Death!” \n",
      "There was immediately a great stir throughout France. Preparations were made for lighting the red bonfires, which would call the country people to rise and fight for King Louis XVI. Above all, at Saint Antoine of the Rue Saint Antoine in Paris there was the greatest tumult; women shrieked, and men swore deep oaths, that Valjean should be thrown into the sea to feed the fishes. It must not be forgotten that Valjean had only been seen by Drouet in the passing crowd of which he gave information. He might easily have made his escape—as he did—in all the hurry and confusion; but instead, he calmly remained where he was, in the heart of Saint Antoine, awaiting his doom.\n",
      "The place was no longer what it had been; the aspect of things was not recognizable. In particular, on that day it bore a terrible expression which it will always be remembered by those who saw it. \n",
      "Saint Antoine was like an armed man, with barricades at all its gates, and red flags and bayonets everywhere. Patrols of twenty-five men each, under the command of some subaltern officer—or sometimes of a municipal guardsman or police sergeant, who could be quickly promoted by the Committee of Public Safety to any military grade from Corporal of a battalion of National Guards to General of Division—paraded its streets in watchful readiness. The inhabitants had set up a barrier with a flag of tricoloured stuff in front of the fountain, and another with a flag of red stuff; red and tricoloured flags were hoisted everywhere else also. Shops and houses were shut up tight on every street where Valjean could have made his way, and the whole district bristled with guns pointed at him from all sides. \n",
      "On the boulevards beyond Saint Antoine, there was a heavy blockade of National Guards; but as they wore their swords drawn, and carried a red flag in addition to their bayonets and muskets, the aspect of those streets was less alarming than that of the interior of Saint Antoine.\n",
      "It happened thus: when Valjean entered Saint Antoine with Javert, he had only just left the Rue du Contrat-Social, where Robespierre and Saint-Just were then holding one of their many meetings in a private room. A man who seemed to have a perfect understanding with the patrols and posts, and whose dress was that of a street porter or messenger, ran up as he entered and touched him on the arm. \n",
      "“Citizen,” said this man, “what you are about to do is dangerous; perhaps useless—who knows? Do you want to take my advice?” \n",
      "Valjean looked at him:\n",
      "“Well! I am a friend; follow me.” He led Valjean through the labyrinth of streets in that quarter, which Valjean did not know\n",
      "llama_print_timings:        load time =   34864.47 ms\n",
      "llama_print_timings:      sample time =     571.87 ms /  1024 runs   (    0.56 ms per token,  1790.61 tokens per second)\n",
      "llama_print_timings: prompt eval time =   11977.42 ms /   494 tokens (   24.25 ms per token,    41.24 tokens per second)\n",
      "llama_print_timings:        eval time =   74930.59 ms /  1023 runs   (   73.25 ms per token,    13.65 tokens per second)\n",
      "llama_print_timings:       total time =   87791.85 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/70B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9aa2443-0fd2-4c90-9bdb-ea4b23eec1f9",
   "metadata": {},
   "source": [
    "### 70B f16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a4e0dd63-3209-4dde-ac88-d73b9e8e7271",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703326874\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 3 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type  f16:  562 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 8192\n",
      "llm_load_print_meta: n_head           = 64\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 80\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 8\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 28672\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 70B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 68.98 B\n",
      "llm_load_print_meta: model size       = 128.48 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.28 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  500.28 MiB\n",
      "llm_load_tensors: VRAM used           = 131065.03 MiB\n",
      "llm_load_tensors: offloading 80 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 81/81 layers to GPU\n",
      "......................................................\n",
      "CUDA error 2 at ggml-cuda.cu:9081: out of memory\n",
      "current device: 0\n",
      "GGML_ASSERT: ggml-cuda.cu:9081: !\"CUDA error\"\n"
     ]
    }
   ],
   "source": [
    "# Out of memory\n",
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/70B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
